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Abstract 


This  report  describes  results  from  two  recent  surveys  conducted  by  the  Software  Engineering 
Institute  (SEI)  to  collect  infonnation  about  the  measurement  and  analysis  activities  of  software 
systems  development  organizations.  Representatives  of  organizations  appraised  at  maturity  levels 
4  and  5  completed  the  survey  in  2008.  Using  a  variant  of  the  same  questionnaire  in  2009,  certified 
high  maturity  lead  appraisers  described  the  organizations  that  they  had  most  recently  coached  or 
appraised  for  the  achievement  of  similar  high  maturity  levels.  The  replies  to  both  surveys  were 
generally  consistent  even  though  the  two  groups  are  often  thought  to  be  quite  different.  The 
results  of  the  surveys  suggest  that  the  organizations  understood  and  used  CMMI-based  process 
performance  modeling  and  related  aspects  of  measurement  and  analysis  a  great  deal.  Both  the 
organizational  respondents  in  2008  and  the  appraisers  in  2009  reported  that  process  performance 
models  were  useful  for  the  organizations. 

The  respondents  in  both  surveys  also  judged  process  performance  modeling  to  be  more  valuable 
in  organizations  that  understood  and  used  measurement  and  analysis  activities  more  frequently  and 
provided  organizational  resources  and  management  support.  In  addition,  results  from  the  2009 
survey  of  lead  appraisers  indicate  that  organizations  that  achieved  their  appraised  high  maturity 
level  goals  also  found  measurement  and  analysis  activities  more  useful  than  those  organizations 
that  did  not  achieve  their  targets. 
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1  Introduction 


1.1  The  Study 

Over  the  past  few  years,  some  critics  have  expressed  concerns  about  the  perfonnance  outcomes 
that  reasonably  can  be  expected  from  CMMI -based  process  improvement.  Clearly  there  are  many 
reasons  why  program  performance  may  be  less  than  optimal,  even  when  the  work  is  done  by  high 
maturity  organizations.  However,  more  and  more  evidence  has  become  available  in  recent  years 
showing  that  CMMI-based  process  discipline  can  result  in  better  program  perfonnance  and 
product  quality,  especially  when  the  work  has  been  done  in  high  maturity  organizations.1 

Still,  additional  clarification  of  high  maturity  practices  is  needed,  along  with  good  examples  of 
what  has  worked  well  and  what  has  not.  This  may  be  particularly  so  with  respect  to  rising 
expectations  about  what  is  valuable  to  organizations  that  achieve  high  maturity  status,  especially 
since  high  maturity  practices  are  so  tightly  coupled  with  measurement  and  analysis.  Moreover, 
appraised  maturity  level  alone  is  not  enough.  We  also  need  to  know  what  kinds  of  measurement 
and  analysis  approaches,  methods,  and  techniques  are  employed  successfully  by  high  maturity 
organizations.  “Success”  should  be  evaluated  using  sound  empirical  evidence  of  better  program 
performance,  product  quality,  and  fitness  for  use. 

These  kinds  of  conversations  need  to  be  conducted  in  a  spirit  of  continuous  improvement.  CMMI 
models  have  matured  since  the  CMM  for  Software  first  appeared,  and  many  more  organizations 
have  achieved  higher  levels  of  capability  and  maturity.2  Nonetheless,  there  is  room  for  better 
understanding  of  how  high  maturity  practices  lead  to  better  project  performance  and  quality 
outcomes  [1], 

A  survey  was  conducted  in  2008  with  organizations  that  attained  CMMI  maturity  levels  4  and  5, 
which  are  commonly  referred  to  as  high  maturity  organizations.  The  questionnaires  were 
completed  by  the  sponsors  of  those  appraisals,  at  times  assisted  by  their  delegates  who  were 
familiar  with  implementation  details  throughout  the  appraised  organizations.  A  variant  on  the 
same  questionnaire  was  used  for  a  2009  survey,  but  it  polled  SEI  High  Maturity  Lead  Appraisers 
(HMLAs). 


1  Some  valuable  references  include  [1]  [2]  [3]  [25]  [26]  [27]  [28]  [29]  [30]  [31]  [32],  Many  more  presentations  and 
articles  have  also  been  published  over  the  past  few  years.  See  especially  recent  presentations  from  the  SEPG 
Conference  and  CMMI  Technology  Conference  series.  Examples  and  quantitative  results  from  somewhat  older 
cases  can  be  found  on  the  CMMI  performance  results  website  at  http://seir.sei.cmu.edu 
/cmmiresearch/results/2005results.asp. 

2  The  SEI  publishes  process  maturity  profiles  that  are  updated  twice  a  year  based  on  the  appraisals  submitted  to 
the  SEI  Appraisal  System.  Profiles  published  since  March  2004  can  be  found  at 

http://www.sei.cmu.edu/cmmi/casestudies/profiles/cmmi.cfm.  Links  to  the  most  recent  versions  of  the  CMMI 
models  for  acquisition,  services,  and  development;  appraisal  methods;  and  other  documents  can  be  found  at 
http://www.sei.cmu.edu/cmmi/tools/index.cfm.  For  comparison,  Version  1.1  of  the  Capability  Maturity  Model  for 
Software  can  be  found  at  http://www.sei.cmu.edu/library/abstracts/reports/93tr024.cfm. 
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Both  surveys  are  part  of  a  larger  SEI  series  that  focuses  on  the  state  of  the  practice  of 
measurement  and  analysis  in  software  and  systems  engineering  organizations.  As  noted  in  the 
2008  high  maturity  survey  report,  similar  results  were  evident  in  two  previous  surveys  in  the 
series  [1]  [2], 3  Based  on  appraised  organizations  across  the  full  spectrum  of  CMMI-based 
maturity  levels,  both  studies  provide  evidence  about  the  usefulness  and  role  of  measurement  and 
analysis  capabilities  to  improve  performance  outcomes. 

Our  intent  was  to  reuse  and  modify  the  original  high  maturity  questionnaire  for  future  surveys, 
using  the  2008  results  as  the  baseline  for  tracking  changes  in  high  maturity  organizations  over 
time.  We  chose  not  to  draw  the  2009  sample  from  the  relatively  small  number  of  organizations 
that  have  achieved  high  maturity  status.  It  does  little  good  to  keep  asking  the  same  people  to 
answer  the  same  questions  each  year.  Instead,  we  surveyed  certified  HMLAs  in  2009.  Doing  so 
provided  useful  comparisons  with  the  perspectives  collected  from  the  appraised  organizations 
themselves. 

In  addition,  the  lead  appraiser  survey  also  allowed  us  to  make  useful  comparisons  among 
organizations  that  could  not  be  made  in  the  survey  of  appraisal  sponsors.  The  sponsor  survey  was 
limited  to  organizations  that  had  been  appraised  at  maturity  levels  4  and  5.  In  contrast,  some  of 
the  organizations  about  which  the  lead  appraisers  reported  in  2009  did  not  achieve  their  appraised 
maturity  level  goals. 

The  questions  in  both  surveys  centered  on  the  use  of  process  performance  modeling  and  the  value 
added  by  its  use.4  By  process  performance  modeling,  we  mean  various  analytical  methods  that 
enable  the  constmction  and  effective  use  of  process  performance  models  and  baselines.  As  was 
true  in  the  earlier  survey,  several  questions  asked  about  the  “healthy  ingredients”  of  a  CMMI- 
based  process  performance  model  [3].  A  list  of  what  the  SEI  considers  healthy  ingredients  can  be 
found  on  page  23. 

Related  questions  asked  about  the  breadth  of  statistical,  experimental,  and  simulation  methods 
used;  attention  paid  to  data  quality  and  integrity;  staffing  and  resources  devoted  to  the  work; 
pertinent  training  and  coaching;  and  the  alignment  of  the  models  with  business  and  technical 
objectives.  Other  questions  asked  about  technical  challenges  and  other  barriers  to  and  facilitators 
of  successful  adoption  and  use  of  measurement  and  analysis. 

Value  added  is  judged  in  both  surveys  by  answers  to  an  identical  series  of  questions  asking  about 
the  results  of  using  process  perfonnance  modeling  (e.g.,  better  product  quality,  fewer  project 
failures,  business  growth,  and  profitability).  Value  added  also  can  be  judged  in  the  lead  appraiser 
survey  by  whether  or  not  the  subject  organizations  achieved  their  appraised  maturity  level  goals. 


3  The  series  began  in  2006  [34],  Comparable  surveys  were  fielded  in  2007,  2008,  and  2009  with  a  focus  on  the 
wider  software  and  systems  engineering  community.  The  intent  is  to  track  change  over  time  when  a  sufficient 
amount  of  data  is  available. 

4  The  questions  used  in  the  two  surveys  are  identical  with  the  exception  of  slight  variations  in  the  wording  to  refer 
to  the  respondents’  own  organizations  in  the  2008  survey  and  the  organizations  with  which  the  HMLAs  worked 
in  the  2009  survey.  There  also  are  a  few  instances  where  questions  were  modified,  added,  or  deleted  for  rea¬ 
sons  of  clarity  and  to  lessen  the  time  burden  on  the  respondents. 
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1 .2  Overview  and  Implications  of  the  Results 

Responses  from  the  appraisal  sponsors  and  the  HMLAs  were  quite  similar.  There  was  a  great  deal 
of  consistency  in  replies  to  the  same  questions,  as  well  as  the  statistical  relationships  among  those 
replies,  even  though  the  two  groups  are  often  thought  to  be  quite  different.  Results  from  both 
surveys  suggest  that  organizations  used  and  had  considerable  understanding  of  process 
performance  modeling  and  related  aspects  of  measurement  and  analysis.  Of  course  the 
respondents’  answers  varied  over  the  range  of  possible  responses  to  the  questions,  but  they  varied 
quite  similarly  in  both  surveys.  Both  groups  also  gave  similar  responses  about  how  useful  process 
performance  models  were  for  the  organizations  about  which  they  were  reporting. 

The  results  from  the  2009  survey  also  suggest  that  variables  such  as  management  support, 
training,  and  the  use  of  statistical  and  other  analytical  methods  were  strongly  related  to  achieving 
the  organizations’  appraised  maturity  level  goals.  The  community  can  be  confident  that  the 
appraisers’  perceptions  are  consistent  with  the  organizations’  views  of  the  value  of  measurement 
and  analysis. 

The  respondents’  judgments  about  the  organizations’  expertise  in  CMMI-based  process 
performance  modeling  were  positively  related  to  their  judgments  about  its  value.  The  HMLAs’ 
judgments  about  the  value  added  by  using  the  PPMs  also  varied  consistently  with  their  judgments 
about  the  organizational  resources  and  management  support  provided  for  that  use.  Room  remains 
for  continuous  improvement  among  high  maturity  organizations;  however,  the  consistency  across 
the  two  surveys  suggests  that  more  widespread  adoption  and  improved  understanding  of  advanced 
measurement  and  analysis  methods  will  likely  improve  CMMI-based  performance  outcomes. 

Some  interesting  differences  do  exist  between  the  results  of  the  two  surveys.  For  example,  as 
shown  in  Section  3,  the  HMLAs  were  slightly  less  positive  in  their  judgments  of  value  added  by 
process  perfonnance  modeling  than  were  the  appraisal  sponsors.  But  appraisers  were  somewhat 
more  likely  to  report  consistent  use  of  process  performance  modeling  approaches  and  analytical 
methods  (Section  4).  The  strength  of  association  among  some  of  the  measures  varies  across  the 
years  (Section  6.1),  and  a  few  noteworthy  differences  are  evident  with  respect  to  achievement  of 
appraisal  target  maturity  level  goals  versus  perceived  value  attributed  to  process  performance 
modeling  (Section  6.22). 

These  two  surveys  are  the  most  comprehensive  of  their  kind  yet  done  and  provide  new 
information  to  the  software  engineering  community.  Our  hope  is  that  they  will  encourage 
additional  applied  research,  resulting  in  better  and  more  widespread  appreciation  of  the  potential 
and  realized  value  for  CMMI-based  process  improvement. 

1.3  What  Follows  in  This  Report 

The  remainder  of  this  document  is  broken  up  into  six  sections  and  three  appendices,  followed  by 
the  references.  A  description  of  the  2009  survey  respondents  and  the  organizations  about  which 
they  reported  is  contained  in  Section  2.  Basic  descriptions  of  variability  in  reported  usefulness  and 
results  of  process  performance  modeling  can  be  found  in  Section  3,  along  with  descriptions  of  the 
achievement  of  appraisal  target  maturity  levels.  Section  4  contains  descriptions  of  process 
performance  modeling  and  related  measurement  and  analysis  methods.  It  also  summarizes  the 
respondents’  answers  to  several  questions  about  the  alignment  and  coordination  of  measurement 
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and  analysis  activities.  The  respondents’  answers  to  questions  about  challenges,  barriers,  and 
facilitators  of  a  successful  measurement  and  analysis  program  are  summarized  in  Section  5. 
Sections  3,  4,  and  5  compare  and  contrast  the  respondents’  answers  to  the  same  questions  asked  in 
the  two  surveys. 

The  extent  to  which  variability  in  process  outcomes  can  be  explained  by  concurrent  differences  in 
process  and  organizational  context  is  examined  in  Section  6.  The  report  summary  and  conclusions 
are  in  Section  7.  The  questions  and  aggregated  responses  to  the  2009  survey  are  reproduced  in 
Appendix  A.5  Appendix  B  contains  free-form  textual  replies  to  the  high  maturity  survey.  These 
provide  a  sense  of  the  meaning  that  can  be  attributed  to  the  quantitative  results.  Appendix  C 
describes  the  analysis  methods  used  in  this  report.  It  is  taken  with  modifications  from  Section  6.2 
of  the  technical  report  that  summarized  the  results  of  the  2008  survey. 

1.4  Methodological  Caveats 

It  is  impossible  to  make  definitive  claims  about  cause  and  effect  based  on  survey  data  such  as 
those  described  in  this  report.  Survey  data  do  not  speak  for  themselves — indeed  no  data  can. 
Interpretations  in  everyday  life  and  in  science  rely  on  experiential  and  theoretical  context  and  a 
thorough  understanding  of  the  measures  themselves.  What  we  describe  in  this  report  is  statistical 
association  at  one  point  in  time.  For  example,  as  shown  in  Section  6.1,  there  is  a  very  strong 
statistical  relationship  between  the  HMLAs’  judgments  about  (1)  the  extent  to  which  process 
performance  model  predictions  are  used  to  inform  decision  making  in  status  and  milestone 
reviews  and  (2)  how  useful  the  process  performance  models  have  been  for  those  same 
organizations.  Similar  results  were  noted  in  the  2008  survey  of  organizations.  It  does  not  matter 
which  of  the  two  surveys  came  first:  what  is  important  is  that  the  two  typically  vary  in  a 
consistent  manner,  reinforcing  each  other. 

In  the  following  sections,  we  make  comparisons  between  appraised  organizations  that  achieved 
their  high  maturity  target  goals  and  those  that  did  not.  We  make  this  distinction  since  evidence 
presented  here  suggests  that  measurement  and  analysis  play  a  significant  role  in  the  organizations’ 
achievement  of  maturity  level  4  or  5.  Again,  we  make  no  claims  about  causation  but  do  note  the 
association.  We  use  all  the  2009  appraisers’  responses  when  making  comparisons  with  the  2008 
survey  since  the  appraisers’  responses  refer  to  organizations  that  felt  ready  to  advance  their 
CMMI  status  to  maturity  levels  4  or  5  and  were  assessed  for  their  high  maturity  capabilities. 

The  goal  of  this  report  is  to  explore  the  data  for  meaningful  associations  among  the  respondents’ 
replies.  We  realize  that  our  interpretations  are  subject  to  our  viewpoint  and  the  limitations  of 
survey  data  in  general.  However,  the  consistency  in  responses  to  many  questions  over  time  and 
across  groups  and  organizations  lends  credence  to  our  interpretations. 


Similar  information  for  the  2008  survey  can  be  found  in  the  earlier  SEI  technical  report  [2], 
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2  The  Respondents  and  the  Appraised  Organizations 


The  2008  survey  of  appraisal  sponsors  and  their  delegates  was  based  on  all  organizations  recorded 
by  the  SEI  as  having  been  appraised  at  CMMI  maturity  levels  4  or  5  over  the  five-year  period 
ending  in  March  2008.  An  invitation  and  up  to  three  reminders  were  sent  in  May  and  June  2008  to 
all  340  high  maturity  organizations.  A  total  of  156  questionnaires  were  returned  for  a  46  percent 
completion  rate. 

Invitations  to  participate  in  the  2009  survey  were  sent  to  all  153  SEI-certified  Standard  CMMI 
Appraisal  Method  for  Process  Improvement  (SCAMPI)  Eligh  Maturity  Lead  Appraisers  (EIMLAs) 
in  March  2009.  Personalized  invitations  and  up  to  three  reminders  to  participate  were  sent  in  April 
and  May.  A  total  of  84  questionnaires  were  returned  for  a  55  percent  completion  rate.  In  both 
surveys,  individual  questions  were  not  answered  by  all  respondents,  resulting  in  different  numbers 
of  responses  (n)  for  different  questions. 

For  the  2009  survey,  EIMLAs  were  asked  to  complete  their  questionnaires  from  the  perspective  of 
the  most  recently  appraised  organizational  unit  they  had  worked  with  that  had  a  goal  of  reaching 
CMMI-based  high  maturity  status.  As  shown  in  Figure  2-1,  over  half  of  the  survey  respondents 
served  as  lead  appraisers  in  their  respective  appraisals  (See  Appendix  A,  Question  1-1).  Fewer  than 
15  percent  served  as  appraisal  team  members,  and  close  to  30  percent  worked  in  a  coaching 
capacity,  helping  organizations  implement  their  high  maturity  practices.6 


One  survey  respondent  did  not  answer  this  question.  The  figures  and  tables  that  follow  in  this  report  always 
include  the  number  of  people  (n)  who  answered  each  question  or  questions. 
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HMLA  Role 


H  Lead  appraiser 
B  Appraisal  team  member 
H  High  maturity  practices  coach 


n=83 


Figure  2-1:  2009  —  Role  ofHMLAs  in  the  appraised  organizations 


The  sample  ofHMLAs  included  those  whose  replies  were  based  on  organizations  that  did  not 
achieve  their  appraised  high  maturity  goals  (See  Appendix  A,  Question  1-8).  This  helps  us  better 
understand  the  HMLAs’  judgments  about  use  and  value  added  by  measurement  and  analysis 
methods  and  related  activities  in  organizations  that  sought  but  did  not  achieve  high  maturity 
status.  According  to  the  appraisers,  over  a  third  of  the  organizations  failed  to  achieve  their 
appraised  high  maturity  level  goals,  although  about  40  percent  of  them  were  close  in  the  judgment 
of  the  HMLAs  who  worked  with  them  (Figure  2-2). 
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Maturity  Level 


H  CMMI  Maturity  Level  3 
or  lower 

■  CloseTo  Maturity  Level  4 


id  CMMI  Maturity  Level  4 


H  CMMI  Maturity  Level  5 


y  Don't  know 


n=82 


Figure  2-2:  2009  —  Appraised  organizations  —  maturity  levels 
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The  types  of  organizations  described  by  the  HMLAs  in  2009  were  generally  similar  to  those 
described  by  the  organizations’  appraisal  sponsors  and  their  delegates  in  2008,  with  the  notable 
exception  of  their  current  maturity  level  status.  The  types  of  organizations  described  by  the 
HMLAs  in  2009  are  summarized  in  Figure  2-3  (See  Appendix  A,  Question  1-3).  More  than  a  third 
of  the  organizational  units  were  government  contractors  or  govermnent  organizations.  That  is 
relatively  close  to  the  29  percent  reported  by  the  sponsors  in  2008.  The  numbers  for  contracted 
new  development  and  in-house  or  proprietary  development  or  maintenance* * 7  reported  by  the 
HMLAs  in  2009  varies  no  more  than  3  percent  from  those  reported  by  the  sponsors  and  their 
delegates  the  previous  year.  The  largest  difference  is  in  the  “other”  category,  where  the  sponsors 
tended  to  make  finer  distinctions  than  did  the  HMLAs.8 


Type  of  organization 

H  Commercial  OTS 

■  Contracted  new 
development 

■  In-house 

H  Defense  contractor 

id  Other  government 
contractor 

H  DoDefense/military 
organization 

U  Other  government 
agency 


Figure  2-3:  2009  —  Sectors  represented  in  the  appraised  organizations 


A  single  category  for  “in-house  or  proprietary  development  or  maintenance”  was  used  in  both  surveys 

The  “other”  category  contained  various  combinations  of  the  rest  of  the  categories,  including  defense,  informa¬ 

tion  technology,  maintenance,  system  integration,  and  service  provision. 
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The  HMLAs’  descriptions  of  the  primary  focus  of  the  work  done  by  the  organizations  are 
summarized  in  Figure  2-4.  Once  again  the  distributions  were  similar.  More  organizations  were 
categorized  by  the  FIMLAs  in  2009  as  being  focused  on  product  or  system  development  and  fewer 
were  described  as  service  providers  than  in  the  sponsors’  descriptions  the  previous  year.  The 
sponsors  again  were  more  likely  to  chose  the  “other”  category  in  response  to  this  question  for  the 
2008  survey  (See  Appendix  A,  Question  1-4). 


Primary  focus  of  work 


B  Product  or  system 
development 

H  Maintenance  or 
sustainment 

HI  Service  provision 
HOther 


n=83 


Figure  2-4:  2009  —  Product  and  service  focus  of  the  organizations 


Both  surveys  had  similar  results  about  the  kinds  of  engineering  that  were  major  parts  of  the 
organizations’  work  and  the  numbers  of  their  full-time  employees  who  worked  predominately  in 
software,  hardware,  or  systems  engineering  (See  Appendix  A,  Question  1-5). 

In  2008,  over  two-thirds  of  the  responding  organizations  were  located  primarily  in  the  United 
States  (27  percent),  India  (29  percent),  and  China  (15  percent).  No  other  country  exceeded  3 
percent  except  Japan  with  4  percent. 
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As  shown  in  Figure  2-5,  40  percent  of  the  organizations  appraised  or  coached  by  the  certified 
FIMLAs  were  located  in  the  United  States.  The  next  largest  number  was  from  China,  which  again 
accounted  for  15  percent  (See  Appendix  A,  Question  1-6). 

The  organizations  surveyed  in  2008  achieved  their  maturity  level  status  over  a  period  of  five  years 
ending  in  March  2008.  The  FIMLAs  were  reporting  about  their  most  recent  engagements  with 
organizations  that  were  (1)  recently  appraised  at  maturity  level  4  or  5,  or  (2)  currently  aspiring  to 
that  status.  Several  newly  certified  FIMLAs  declined  to  participate  in  the  survey  because  they  had 
not  yet  had  a  coaching  or  appraisal  engagement  since  being  certified. 


Location 


H  United  States 
H  China 
Q  India 
H Japan 
EJ  All  Others 


n=82 


Figure  2-5:  2009  —  Primary  location  of  the  organizations 


10  I  CMU/SEI-201 0-TR-022 


3  A  Focus  on  Overall  Value  and  Appraisal  Results 


A  major  purpose  of  the  high  maturity  measurement  and  analysis  surveys  is  to  find  out  what 
measurement  and  analysis  resources  exist  and  what  activities  occur  in  organizations  that  have 
achieved  high  maturity  status.  The  development  of  such  models  is  discussed  in  the  CMMI 
Organizational  Process  Performance  (OPP)  process  area,  and  the  varied  uses  of  process 
performance  models  are  covered  in  the  Quantitative  Project  Management  (QPM),  Causal  Analysis 
and  Resolution  (CAR),  and  Organizational  Innovation  and  Deployment  (OID)  process  areas. 
Numerous  studies  have  described  failures  of  measurement  programs  in  software  development 
organizations  [4]  [5]  [6]  [7]  [8]  [9].  Our  sample  is  skewed  in  the  other  direction:  all  of  the  high 
maturity  organizations  surveyed  in  2008  or  appraised  for  high  maturity  status  in  the  2009  survey 
had  relatively  successful  measurement  programs. 

A  focus  on  the  value  of  implementing  successful  measurement  and  analysis  programs  provides 
examples  of  the  kinds  of  analytical  resources  and  activities  successful  organizations  employ.  We 
are  aware  that  confusion  exists  about  what  is  necessary  to  meet  the  goals  of  a  CMMI-based 
process  improvement  appraisal.  In  recent  years,  for  example,  much  attention  has  been  focused  on 
what  constitutes  a  process  performance  model.  The  purpose  of  this  report  is  to  address  these 
issues  and  provide  better  guidance  to  the  community  by  showing  what  measurement  and  analysis 
activities  lead  appraisers  have  reported  for  organizations  seeking  high  maturity  status.  These  data, 
coupled  with  data  from  the  2008  survey  of  high  maturity  organizations,  show  that  measurement 
and  analysis  activities  have  notable  perceived  business  value  to  the  organizations.  The  2009 
results  show  that  these  activities  also  impact  the  achievement  of  the  organization’s  high  maturity 
level  goals.  This  section  presents  the  results  of  the  survey  responses  for  questions  pertaining  to 
usefulness  or  value  of  process  performance  models. 

3.1  Usefulness  of  Process  Performance  Models  and  Achievement  of  Appraisal  Target 
Maturity  Level 

As  in  the  2008  organizational  survey,  we  asked  the  HMLAs  “how  useful  have  process 
performance  models  been  for  this  organization  overall?”  (See  Appendix  A,  Question  VI-5)  We 
asked  the  appraisers  to  choose  from  the  following  alternatives: 

•  Extremely  valuable  -  they  rely  on  them  extensively 

•  Very  valuable  -  they  have  obtained  much  useful  information  from  them 

•  Mixed  value  -  they  have  obtained  useful  infonnation  on  occasion 

•  Little  or  no  value 

•  Harmful,  not  helpful 

•  Don’t  know 
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The  range  of  their  responses  and  the  organizational  representatives’  replies  to  a  comparable 
question  in  the  2008  survey  are  shown  in  Figure  3-1°  (see  Appendix  A,  Question  VI-5).  The 
perspectives  of  the  organizations  surveyed  in  2008  are  very  similar  to  those  of  the  HMLAs 
surveyed  in  2009  in  their  perception  of  the  usefulness  of  process  performance  models.10 


In  your  best  judgment,  how  useful  have 
process  performance  models  been  for  this 
organization  overall? 


U  2008 (n  =  144) 
H  2009  (n  =  75) 


Little/No  Value  Mixed  value  Very  valuable  Extremely 

valuable 


Figure  3-1:  2008  &  2009  -  Usefulness  of  process  performance  models 


No  respondents  selected  “Harmful,  not  helpful”  in  either  survey.  One  respondent  chose  “Don't  know”  in  2009. 

10  We  can  use  the  statistical  test  of  proportions  to  affirm  statements  of  “similar”  or  “different”  for  individual  pairs  of 
2008  and  2009  category  data.  In  the  above  case,  each  category  of  response  has  a  pair  of  proportions 
representing  the  2008  and  2009  samples.  For  example,  the  category  “Very  Valuable”  has  74  (of  144)  respon¬ 
dents  from  2008  and  34  (of  75)  respondents  from  2009.  The  corresponding  test  uses  a  null  hypothesis  that  the 
proportions  are  equal  versus  the  alternative  hypothesis  that  they  are  not  equal.  Fisher’s  exact  test  calculates  a 
p-value  of  0.477,  which  fails  to  reject  the  null  hypothesis.  All  four  categories  in  Figure  3-1  show  no  statistical  dif¬ 
ference  in  proportion.  We  used  Fisher’s  exact  test  since  it  can  be  applied  to  any  size  sample.  Throughout  this 
report  we  make  comparisons  but  in  the  interest  of  brevity  do  not  present  the  underlying  statistical  test  which  jus¬ 
tifies  a  particular  statement.  The  reader  may  use  the  data  available  in  Appendix  A  to  perform  most,  if  not  all,  of 
these  background  statistical  tests. 
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Nearly  60  percent  in  both  years  perceived  the  overall  usefulness  of  process  performance  models 
as  very  or  extremely  valuable.  About  5  percent  derived  little  value  or  no  value  from  process 
performance  models.  An  additional  question  in  the  2009  survey  divided  the  appraised 
organizations  into  those  that  achieved  their  maturity  target  goal  versus  those  that  did  not.  When 
applied  to  the  2009  data  in  Figure  3-1,  it  shows  that  deriving  increased  value  from  process 
performance  models  is  associated  with  an  organization  achieving  its  high  maturity  target  goal 
from  a  CMMI  appraisal  (Figure  3-2). 


Figure  3-2:  2009  —  Relationship  between  achievement  of  appraised  high  maturity  level 

targets  and  overall  value  attributed  to  process  performance  models 


The  responses  clearly  indicate  a  difference  in  valuation  of  process  performance  models  for  those 
organizations  that  achieved  their  high  maturity  goal  versus  those  that  did  not.  More  than  70 
percent  of  those  organizations  that  achieved  their  goals  found  process  performance  models  very  or 
extremely  valuable,  while  about  75  percent  of  the  other  group  found  mixed  or  little  value.  This 
distinction  is  confirmed  statistically  by  the  Mann- Whitney  test,  which  demonstrates  a  significant 
difference  between  the  two  groups  (p  <  .002)  [10]  [11]."  These  results  suggest  several  hypotheses 
about  the  relationships  between  response  variables  and  form  a  basis  for  much  of  the  analyses 
presented  in  Section  6.  This  insight  also  suggests  implications  about  value  that  are  not  apparent  by 
surveying  only  successful  organizations  as  we  did  in  2008. 


11  We  used  the  Mann-Whitney  test  to  perform  a  hypothesis  test  of  the  equality  of  two  population  medians.  The  test 
requires  that  the  two  samples  have  equal  variances,  which  we  also  tested  using  Levene’s  test  for  equal  va¬ 
riances.  Whenever  we  present  a  Mann-Whitney  test  result,  we  also  performed  Levene’s  test.  Here  and  for  later 
reported  results,  Levene’s  test  failed  to  reject  the  null  hypothesis  that  the  variances  were  equal. 
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3.2  Results  of  Process  Performance  Modeling 


The  respondents  were  asked  questions  that  described  several  specific  results  experienced  by  high 
maturity  organizations  using  process  performance  modeling  (see  Appendix  A,  Question  VI-3). 
Figure  3-3  shows  that  more  than  three-quarters  of  the  HMLAs  reported  better  project  performance 
or  product  quality  frequently  or  almost  always.  Fewer  project  failures  were  reported  by  over  60 
percent  of  the  respondents  as  frequent  or  almost  always.  Almost  as  many  respondents  for  those 
categories  reported  better  tactical  decision  making,  and  over  40  percent  said  they  experienced 
improvements  in  strategic  decision  making  as  a  result  of  their  process  performance  modeling 
activities. 


2009:  Statements  about  the  possible  results  of  using 
process  performance  modeling 
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Figure  3-3:  2009  —  Results  of  using  process  performance  models 
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These  results  from  the  appraisers  closely  mirror  the  results  obtained  from  organizations  surveyed 
in  2008,  as  shown  in  Figure  3-4. 12 


2008:  Statements  about  the  possible  results  of  using 
process  performance  modeling 
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Figure  3-4:  2008  —  Results  of  using  process  performance  models 


12  Again,  we  used  the  test  of  proportions  here  after  combining  the  two  response  categories  “Almost  always”  and 
“Frequently.”  The  null  hypothesis  of  no  difference  was  not  rejected  for  each  of  the  five  variables  discussed. 
There  are  differences  between  Figures  3-3  and  3-4  in  the  finer  distinctions  of  the  categories,  but  they  are  not  a 
focus  of  this  report. 
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Another  factor  contributing  to  the  value  added  by  process  performance  models  is  the  use  of 
modeling  outputs  to  verify  or  adjust  project  resources  and  effort  during  development.  In  both 
years,  we  asked  if  process  performance  model  predictions  were  used  to  inform  decision  making  in 
the  organization’s  status  and  milestone  reviews  (see  Appendix  A,  question  VI-4).  The  responses 
shown  in  Figure  3-5  show  that  60  percent  of  both  the  appraiser  and  organizational  respondents 
almost  always  or  frequently  used  the  model  predictions  (estimates)  as  information  for  making 
decisions  at  status  and  milestone  reviews.  The  2008  responses  indicate  that  21  percent  of  the 
organizations  use  such  information  only  occasionally  or  rarely,  while  in  2009  appraisers  indicated 
3 1  percent.  This  question  assumes  that  the  models  in  use  have  a  proven,  high  degree  of  accuracy 
and  suitability. 


How  often  are  process  performance  model  predictions  used  to  inform 
decision  making  in  the  organization's  status  and  milestone  reviews? 


45% 


Almost  always  Frequently  About  half  the  Occasionally  Rarely  if  ever 

time 


a  2008 
(n=142) 


H  2009 
(n=71) 


Figure  3-5:  2008  &  2009  -  Use  of  process  performance  models  to  inform  reviews 
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This  use  of  process  performance  models  is  reflected  in  whether  or  not  the  organization  appraised 
achieved  its  maturity  level  target,  as  shown  in  Figure  3-6.  The  Mann- Whitney  test  confirmed  a 
difference  between  the  two  groups  (p  <.007).  Appraisers  reported  that  73  percent  of  organizations 
that  achieved  their  goal  used  process  performance  model  predictions  frequently  or  almost  always 
versus  38  percent  of  those  organizations  that  did  not  achieve  their  goal.  Conversely,  only  25 
percent  of  the  achiever  group  seldom  used  process  perfonnance  model  predictions  in  this  manner, 
versus  53  percent  of  the  non-achiever  group. 
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Figure  3-6:  2009  -  Use  of  process  performance  models  to  inform  reviews  related  to 

achieving  appraisal  target 

In  this  and  other  instances  presented  here,  there  are  statistically  significant  differences  that  are 
consistent  with  our  expectations  about  the  performance  of  high  maturity  organizations.  However, 
when  conducting  many  statistical  tests  on  a  sample,  a  small  number  of  comparisons  are  bound  to 
be  significant  by  chance  [12].  Hence,  we  only  report  p-values  when  we  are  testing  a  few  specific 
hypotheses  as  part  of  our  exploratory  data  analysis. 

The  results  presented  in  this  section  form  a  compelling  story  that  links  the  usefulness  and  value  of 
using  process  performance  models  in  development  organizations.  We  can  see  from  the  2008  and 
2009  surveys  that  the  organizations  and  appraisers  viewed  this  usefulness  similarly.  When  we 
distinguish  those  organizations  that  failed  to  achieve  their  appraisal  target  in  the  2009  survey,  we 
also  see  that  the  association  of  use  and  value  is  quite  pronounced. 
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4  Measurement  and  Analysis  Activities  and  Techniques 


Both  the  2008  and  2009  surveys  solicited  information  about  the  analytical  techniques  used  by  the 
organizations.  The  results  of  these  surveys  may  be  useful  to  organizations  seeking  insight  into 
measurement  and  analysis  practices  and  techniques,  whether  or  not  they  are  trying  to  achieve  high 
maturity  status.  This  information  is  also  likely  to  be  valuable  to  improvement  consultants  and 
appraisers  who  suggest  or  evaluate  high  maturity  practices.  The  previous  section  showed  the 
respondents’  views  about  the  usefulness  and  value  of  process  perfonnance  models;  this  section 
summarizes  their  responses  to  several  additional  questions  about  the  specific  uses  of  the  process 
performance  models,  related  analytical  methods,  and  measurement  and  analysis  activities. 

4.1  Use  of  Process  Performance  Models  and  Baselines 

Figure  4-1  shows  considerable  variation  perceived  by  respondents  in  the  routine  use  of  process 
performance  models  (See  Appendix  A,  Question  IV-1).  This  variation  is  not  surprising  given  the 
many  domains  and  application  areas  encompassed  by  the  organizations  involved,  and  each 
organization  also  had  its  own  business  goals  and  customer  requirements.  Delivered  defect 
prediction  was  reported  as  commonly  modeled  by  over  80  percent  of  the  organizations  in  both 
surveys.  As  expected,  cost  and  schedule  prediction  were  widely  used,  while  other  outcomes  were 
predicted  less  often.  Return  on  investment  (ROI)  from  process  improvement  was  modeled  least 
often,  although  still  by  over  20  percent  of  the  organizations  in  both  surveys. 

Figure  4-1  also  illustrates  another  important  aspect:  the  similarity  of  perception  across  the  two 
surveys.  Certified  HMLAs  are  generally  in  aligmnent  with  the  organizations  when  it  involves 
these  topics.  When  deviations  do  exist  between  specific  appraiser  and  organizational  responses, 
we  can  only  conjecture  that  the  appraisers  are  evaluating  artifacts  while  the  organizations  are 
evaluating  performance  outcomes.  Organizational  representatives  are  also  more  likely  to  be 
familiar  with  pertinent  details  and  day-to-day  operations. 

Figure  4-2  illustrates  which  processes  and  activities  were  routinely  modeled  (See  Appendix  A, 
Question  IV-3).  Seventy  percent  of  organizations  indicated  that  software  design  and  coding  were 
modeled,  closely  followed  by  project  planning/estimation  and  quality  control  processes. 
Appraisers  ranked  project  planning/estimation  first,  followed  by  quality  control  processes  and  the 
software  design  and  coding.  Modeling  for  requirements  engineering  also  found  wide  adoption, 
while  in  the  other  areas  organizations  used  models  less  often. 

Figure  4-3  shows  the  process  performance  outcomes  that  were  routinely  predicted  (See  Appendix 
A,  Question  IV-2).  Responses  from  both  years  indicate  high  levels  of  usage  for  estimates  at 
completion,  escaped  defects,  effectiveness  or  efficiency  of  inspection  or  test  coverage,  and  cost  of 
quality  and  poor  quality  (e.g.,  rework). 
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Which  of  the  following  product  quality  and  project  performance  outcomes  were 
routinely  predicted  with  process  performance  models  in  the  organization? 
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Figure  4-1:  Quality  and  performance  predictions 
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Which  of  the  following  processes  and  activities  were  routinely  modeled  in  the  organization? 
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Figure  4-2:  Modeling  of  processes  and  activities 
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Which  of  the  following  (often  interim)  process  performance  outcomes  are 
routinely  predicted  with  process  performance  models  in  your  organization? 
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Figure  4-3:  Modeling  of  process  outcomes 
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Over  the  past  three  years,  SEI  technical  staff  and  others  have  worked  to  clarify  what  have  been 
called  the  “healthy  ingredients”  of  CMMI-based  process  performance  models  [3]  [13].  These 
“ingredients”  comprise  a  set  of  analytical  activities  and  uses  of  data.  Models  that  include  the 
healthy  ingredients 

1 .  are  statistical,  probabilistic,  or  simulation  in  nature 

2.  predict  interim  and/or  final  project  outcomes 

3.  use  controllable  predictive  factors  directly  tied  to  sub  processes  or  work  activities 

4.  quantitatively  characterize  and  model  the  variation  of  the  predictive  factors  and  understand 
the  predicted  range,  uncertainty,  or  variation  of  the  outcome  performance  measures 

5.  enable  what-if  analysis  for  project  planning,  dynamic  re-planning,  and  problem  resolution 
during  project  execution 

6.  connect  upstream  activity  with  downstream  activity 

7.  enable  projects  to  achieve  mid-course  corrections  to  ensure  project  success 

In  both  surveys,  two  series  of  questions  focused  on  the  detailed  aspects  of  these  ingredients. 
Presented  later  in  this  report,  composite  measures  based  on  answers  to  both  series  of  questions 
showed  a  strong  association  with  respondents’  reports  about  the  value  of  using  process 
performance  models. 

The  first  set  of  questions  (see  Appendix  A,  Question  IV-4)  focused  on  the  emphasis  the 
respondents  placed  on  the  various  healthy  ingredients  in  their  modeling  efforts.  The  results  in 
Figure  4-4  show  that  the  organizations  varied  considerably  in  their  emphasis  on  these  factors. 
More  than  half  of  the  organizations  placed  an  emphasis  on  modeling  for  uncertainty/variability, 
controllable  factors,  and  detailed  sub  processes. 

In  the  second  set  of  questions  about  the  healthy  ingredients  of  a  CMMI-based  process 
performance  model  (see  Appendix  A,  Question  IV-5),  we  asked  about  the  purposes  for  which  the 
organizations  used  their  models.  Figure  4-5  shows  that  appraisers  saw  widespread  use  of  process 
performance  models  for  predicting  project  final  and  interim  outcomes,  modeling  variation  to 
make  mid-course  corrections,  and  to  enable  contingency  (i.e.,  “what-if’)  analyses. 
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How  much  emphasis  does  the  organization  place  upon  the  following  in  its  process 

performance  modeling? 
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Figure  4-4:  2009  —  Emphasis  of  process  performance  modeling 
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To  what  degree  are  your  organization's  process  performance  models 
used  for  the  following  purposes? 
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Figure  4-5:  2009  —  Purpose  of  process  performance  modeling 
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4.2  Use  of  Other  Analytical  Methods  and  Techniques 


In  both  surveys,  we  asked  about  specific  analytical  techniques  and  methods  in  use  at  the 
organizations.  Figure  4-6  shows  the  results  for  the  first  question  in  the  2009  survey  that  asked 
about  the  extent  of  use  of  the  following  statistical  techniques  (see  Appendix  A,  Question  V-l): 


•  Regression  analysis  predicting  continuous  outcomes  (e.g.,  bivariate  or  multivariate  linear 
regression  or  non-linear  regression) 

•  Regression  analysis  predicting  categorical  outcomes  (e.g.,  logistic  regression  or  loglinear 
models) 

•  Analysis  of  variance  (e.g.,  ANOVA,  ANCOVA,  or  MANOVA) 

•  Attribute  statistical  process  control  (SPC)  charts  (e.g.,  c,  u,  p,  or  np) 

•  Individual  point  SPC  charts  (e.g.,  ImR  or  XmR) 

•  Continuous  SPC  charts  (e.g.,  XbarR  or  XbarS) 

•  Design  of  experiments 

•  Other 


To  what  extent  are  the  following  statistical  methods  used  in 
the  organization's  process  performance  modeling? 
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Figure  4-6:  2009  —  Use  of  statistical  methods  in  process  performance  models 
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Individual-point  SPC  charts  were  most  widely  used,  closely  followed  by  regression  for  continuous 
outcomes,  and  then  analysis  of  variance  and  attribute  SPC  charts.  Few  organizations  reported 
extensive  or  substantial  use  of  discrete  regression  or  design  of  experiments,  although  many 
organizations  did  report  moderate  use  of  those  techniques.  We  also  formed  a  composite  weighted 
summed  index  of  all  these  methods.  A  comparison  with  the  2008  results  is  shown  in  Figure  4-7. 
Again,  both  the  appraisers  and  the  organizations  gave  very  similar  responses.13 

To  what  extent  are  statistical  methods  used  in 
the  organization's  process  performance 
modeling? 
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Figure  4-7:  2008  &  2009  -  Use  of  statistical  methods  in  process  performance  models 


13  The  Chi-Square  test  fails  to  reject  the  null  hypothesis  that  there  is  no  difference  between  2008  and  2009  data 
(p-value  =  0.601).  This  test  is  a  good  indication  that  the  overall  profiles  are  similar. 
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A  different  picture  emerges  when  we  consider  the  2009  data  based  on  whether  or  not  the 
appraised  organizations  achieved  their  high  maturity  level  goals  (Figure  4-8).  We  see  a  clear 
association  of  achieving  a  targeted  maturity  level  with  moderate  to  extensive  use  of  this  variety  of 
statistical  methods.  Again,  the  Mann- Whitney  test  shows  this  distinction  between  the  two  groups 
(p  c.001). 
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Figure  4-8:  2009  —  Use  of  statistical  methods  related  to  achieving  appraisal  target 

We  also  asked  about  the  use  of  several  optimization  and  simulation  techniques  (see  Appendix  A, 
Question  V-3).  These  methods  are  most  useful  in  organizations  that  have  successfully 
implemented  process  performance  models  because  the  most  efficient  parameter  settings  can  be 
derived  based  on  specified  conditions  included  in  the  process  performance  models. 

The  set  of  responses  included 

•  Monte  Carlo  simulation 

•  Discrete  event  simulation  for  process  modeling 

•  Markov  or  Petri-net  models 

•  Probabilistic  modeling 

•  Neural  networks 

•  Optimization 

•  Other 

•  None  of  the  above 
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Figure  4-9  presents  the  results  for  both  the  2008  and  2009  surveys.  Monte  Carlo  simulation 
proved  to  be  the  most  widely  used  optimization  method,  employed  by  over  40  percent  of  the 
organizations  in  both  surveys. 


Which  of  the  following  other  optimization  approaches  are  used  in  your  organization's 

process  performance  modeling? 


Monte  Probabilistic  Optimization  Discrete  None  Don't  Markov  Other  Neural 

Carlo  modeling  event  ofthe  know  or  networks 

simulation  simulation  above  Petri-net 

models 

H  2008  Orgs  (n=143) 

H  2009  HMLAs  (n=78) 


Figure  4-9: 


2008  &  2009 —  Use  of  optimization  techniques  in  process  performance  models 
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Figure  4-10  shows  responses  to  a  question  in  the  2009  survey  about  the  visual  display  techniques 
used  when  performing  analytical  tasks  (see  Appendix  A,  Question  V-2).  Pareto  charts,  pie  charts, 
and  bar  charts  are  widely  used,  followed  by  histograms  and  scatterplots. 


2009:  Use  of  Visual  display  techniques 
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Figure  4-10:  2009  —  Use  of  visual  techniques  with  process  performance  models 
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Figure  4-1 1  shows  responses  to  a  question  in  the  2009  survey  about  the  use  of  various  decision 
techniques  (see  Appendix  A,  Question  V-4).  Weighted  multi-criteria  methods  (e.g.,  QFD  or 
Pugh),  wide-band  Delphi,  and  decision  trees  were  the  most  common  methods  seen  by  appraisers. 
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Figure  4-11:  2009  —  Use  of  decision  techniques  in  process  performance  models 

4.3  Alignment  and  Coordination  of  Measurement  and  Analysis  Activities 

The  importance  of  stakeholder  involvement  is  widely  acknowledged  for  process  improvement  in 
general  and  measurement  and  analysis  in  particular.  This  can  be  seen  in  CMMI  generic  practice 
2.7,  which  emphasizes  the  importance  of  identifying  and  involving  relevant  stakeholders  during 
process  execution.  Stakeholder  involvement  is  also  important  throughout  the  life  cycle.  Such 
notions  are  basic  to  goal-driven  measurement  [14]  [15],  It  also  is  crucial  for  the  CMMI 
Measurement  and  Analysis  process  area,  particularly  in  specific  goal  1  and  specific  practice  1.1, 
which  are  meant  to  ensure  that  measurement  objectives  and  activities  are  aligned  with  the 
organizational  units’  information  needs  and  objectives.  Specific  practice  2.4  also  emphasizes  the 
importance  of  reporting  the  results  to  all  relevant  stakeholders.  Empirical  evidence  indicates  that 
the  existence  of  such  support  increases  the  likelihood  of  the  success  of  measurement  programs  in 
general  [5]  [8]  [16]. 
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We  asked  a  series  of  questions  seeking  feedback  on  the  involvement  of  eight  different  categories 
of  potential  stakeholders  (Figure  4-12).  Measurement  specialists  have  the  largest  proportion  of 
involvement  at  70  percent,  closely  followed  by  process/quality  engineers  and  executives/senior 
management.  A  large  number  of  organizations  also  showed  substantial  involvement  by  middle 
managers  and  project  managers  (see  Appendix  A,  Question  III-l). 
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How  would  you  characterize  the  involvement  of  various 
potential  stakeholders  in  setting  goals  and  deciding  on  plans 
of  action  for  measurement  and  analysis  in  the  organization? 


Customers  Executive  Middle  Project  Project  engineers/  Process  Measurement 

n=71  and  senior  managers  managers  technical  and  quality  specialists 

managers  n=76  n=76  staff  engineers  n=70 

n=75  n=72  n=75 


Figure  4-12:  2009  —  Stakeholder  involvement  in  organization  measurement  and  analysis 

We  asked  appraisers  to  describe  how  staffing  for  measurement  and  analysis  activities  varied 
across  organizations  (see  Appendix  A,  Question  III-2).  An  organization-wide  group,  a  division,  or 
a  similar  corporate  support  group  (e.g.,  an  engineering  process,  quality  assurance,  or  measurement 
group)  was  identified  by  50  of  78  respondents.  Separate  groups  or  individuals  in  different  projects 
or  other  organizational  units  (e.g.,  project,  product  team,  or  similar  work  groups)  were  seen  in  25 
organizations,  while  24  had  a  few  key  people  who  were  measurement  experts  (or  one  person). 
Other  staffing  arrangements  were  seen  in  seven  organizations.  These  arrangements  were  not 
necessarily  mutually  exclusive:  23  of  the  78  respondents  reported  more  than  one  type. 


H  Little  if  any 
H  Limited 
B  Moderate 
■  Substantial 
B  Extensive 
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Properly  done,  measurement-related  training  and  coaching  can  cultivate  the  expertise  needed  for 
the  use  of  process  performance  models.  Figure  4-13  summarizes  the  appraisers’  characterizations 
of  such  training  in  the  organizations  about  which  they  reported  (see  Appendix  A,  Question  II- 1). 
Good  or  excellent  training  was  particularly  widespread  for  coaches,  process/quality  engineers, 
process  perfonnance  model  builders,  data  handlers,  and  project  managers. 


How  would  you  best  characterize  the  measurement  related  training 
that  is  available  in  the  organization? 
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Figure  4-13:  2009  -  Measurement-related  training 
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Beyond  training,  good  understanding  of  the  statistical  results  of  modeling  forms  a  basis  for 
improved  decision  making.  For  the  organizations  that  used  process  performance  model  results,  we 
asked  how  well  the  managers  understood  those  results  (see  Appendix  A,  Question  II-4).  Figure 
4-14  shows  almost  40  percent  of  managers  were  judged  to  understand  such  results  very  or 
extremely  well.  This  suggests  that  management  training  could  increase  the  benefits  gained  from 
using  measurement  and  analysis  results  to  inform  decision  making. 


How  well  do  the  managers  in  the 
organization  who  use  process  performance 
model  results  understand  the  results  that 
they  use? 


H  Hardly  at  all 
H  To  some  extent 
H  Moderately  well 
H  Very  well 
H  Extremely  well 


Figure  4-14: 


2009  -  Managers  ’  understanding  of  process  performance  model  results 
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Of  course  high-quality  training  alone  does  not  ensure  success.  Capable  staff  must  be  available 
when  needed  to  do  the  work,  provide  consultation  and  advice,  and  answer  questions  that  arise. 
Figure  4-15  shows  that  the  organizations  provided  the  necessary  expertise  on  a  timely  basis,  with 
more  than  half  making  qualified  people  available  more  than  60  percent  of  the  time  (see  Appendix 
A,  Question  II-5). 


How  often  are  qualified,  well-prepared 
people  available  to  work  on  process 
performance  modeling? 


H  Rarely,  if 
ever 

■  Occasionally 


B  About  half 
thetime 

B  Frequently 


B  Almost 
always 


Figure  4-15:  2009  —  Availability  of  qualified  personnel 


When  an  organization  attempts  to  attain  higher  maturity  status,  the  need  to  incorporate  analytical 
decision  making  into  the  development  culture  also  means  that  the  people  who  create  process 
performance  models  must  understand  the  specific  goals  of  their  modeling  efforts  and  the  ways  in 
which  the  results  can  be  used.  Process  perfonnance  models  built  to  further  process  and  product 
improvement  goals  also  require  thorough  knowledge  of  the  baseline  performance  to  enable 
quantitatively  informed  judgments  about  the  effects  of  the  modeled  changes.  We  asked  the 
FIMLAs  to  judge  how  well  the  people  who  actually  create  the  models  and  baselines  understood 
the  intent  of  CMMI  (see  Appendix  A,  Question  II-3). 
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Appraisers  saw  considerable  understanding,  with  over  half  judged  to  understand  very  or 
extremely  well  what  constitutes  a  CMMI-based  process  performance  model  or  process 
performance  baseline  and  the  circumstances  where  they  are  useful  (Figure  4-16). 


How  well  do  the  people  who  create  the 
organization's  process  performance  models  and 
baselines  understand  the  intent  of  CMMI? 


■  Hardly  at  all 
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■  Very  well 

H  Extremely  well 
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PPM  PPM  baseline  when  PPM  when  PPMs  are 
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Figure  4-16:  2009  —  Process  performance  model  personnel’s  understanding  of  CMMI 
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Finally,  an  important  aspect  of  an  organization’s  measurement  and  analysis  infrastructure  is  the 
consistent  and  accessible  documentation  of  data,  including  its  generation,  collection,  storage,  and 
use.  Figure  4-17  shows  that  from  the  perspectives  of  the  high  maturity  lead  appraisers,  66  percent 
of  these  organizations  provided  good  or  excellent  documentation  of  results. 


How  would  you  best  describe  the 
organization's  documentation  of  its  process 
performance  and  quality  measurement 
results? 


H  Exclient 

■  Good 

H  Adequate 
B  Fair 

■  Poor 


n=79 


Figure  4-17:  2009  —  Quality  of  documentation  of  measured  results 
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The  appraisers’  judgments  about  the  organizations’  documentation  of  their  process  performance 
and  quality  measurement  results  differ  markedly  for  those  organizations  that  achieved  their 
appraisal  goals  versus  those  that  did  not.  Figure  4-18  shows  this  distinction  which  is  confirmed  by 
the  Mann- Whitney  test  (p  <  .0000). 


2009:  Describe  the  Organization's  Documentation  of  its' 
Process  Performance  and  Quality  Measurement  Results 


Poor  Fair  Adequate  Good  Excellent 


H  Did  Not 
Achieve 
MLGoal 
(n=23) 

H  Achieved 
MLGoal 
(n=44) 


Figure  4-18:  2009  —  Documentation  of  measured  results 
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5  Challenges,  Barriers,  and  Facilitators  of  Successful 
Measurement  and  Analysis 


5.1  Technical  Challenges 

We  asked  the  respondents  about  a  series  of  technical  challenges  that  the  projects  and  product 
teams  in  their  organizations  might  have  faced  (see  Appendix  A,  Question  VI- 1).  The  questions 
focused  most  heavily  on  product  characteristics,  although  some  related  more  directly  to 
organizational  context.  The  concern  was  that  different  degrees  of  technical  challenge  warrant 
different  degrees  of  sophistication  in  measurement,  analytic  methods,  and  predictive  modeling. 
Previous  work  suggests  that  such  challenges  can  directly  affect  the  chances  of  project  success 
independently  of  project  capability  [17]  [18]. 

Figure  5-1  shows  the  responses  for  both  2008  and  2009.  Generally,  the  organization  group  saw  a 
greater  prevalence  of  technical  challenges,  especially  in  the  areas  of  requirements,  size  of  the 
development  effort,  interoperability  needs,  and  constraints  on  product  quality  attributes  (e.g., 
reliability,  scalability,  security,  and  supportability).  This  is  not  surprising  since  the  appraisers  are 
less  likely  to  be  familiar  with  these  kinds  of  issues  than  are  the  organizations  themselves. 
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Following  are  a  series  of  statements  about  the  kinds  of  technical  challenges  that  projects  sometimes 

face.  Which  ones  typically  describe  this  organization? 

100% 

90% 


80% 


Requirements  Size  Interoperability  Requirements  Constraints  Insufficient  Technology  Little 

change  needs  not  well  on  product  resources  not  mature  or  no 

defined  quality  precedent 


Insufficient 

expertise 


Other 


U  2008  Orgs  (n=137) 
H  2009  L.A.  (n=78) 


Figure  5-1:  2008  &  2009  —  Technical  challenges 
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5.2  Obstacles 


A  series  of  questions  asked  appraisers  in  2009  and  organizations  in  2008  about  major  obstacles 
that  might  have  inhibited  progress  of  the  organization’s  journey  to  high  maturity  (see  Appendix 
A,  Question  VII- 1).  As  shown  in  Figure  5-2,  several  of  these  potential  difficulties  were 
experienced  by  20  percent  or  more  of  the  organizations,  and  about  20  percent  reported  not  having 
experienced  any  of  them.  Many  of  the  questions  in  this  series  overlap  with  others  in  the  survey, 
and  others  are  only  loosely  coupled  to  process  performance  modeling. 

The  most  common  problem  (identified  by  45  percent  of  respondents)  was  the  failure  to  collect 
enough  contextual  information  for  proper  segmentation  and  stratification,  meaning  data  could  not 
be  categorized  or  grouped  for  useful  analysis.  The  second  most  common  obstacle  (identified  by 
36  percent  of  respondents)  was  a  failure  to  achieve  enough  consistency  in  their  measures  to 
aggregate  and  disaggregate  them  properly  across  the  organization.  Without  sufficient 
commonality  across  organizational  units,  data  could  not  be  usefully  grouped.  These  types  of 
problems  speak  to  a  lack  of  understanding  of  the  value  of  having  common  measurement 
definitions,  procedures  for  data  handling,  and  shared  knowledge  of  how  measures  map  to  business 
goals  in  the  organization. 

Figure  5-2  also  shows  that  the  appraisers  and  organizations  differed  somewhat  when  ranking 
major  obstacles.  The  organizational  representatives  in  2008  rated  resistance  to  collecting  new  or 
additional  data  after  achieving  maturity  level  3,  lack  of  access  by  process  performance  modelers 
to  people  with  sufficient  statistical  expertise,  and  data  not  being  collected  frequently  enough  to 
make  mid-course  corrections  slightly  higher  than  did  the  appraisers  in  2009.  On  the  other  hand, 
the  HMLAs  tended  to  focus  somewhat  more  on  the  technical  issues  of  process  performance 
modeling,  particularly  the  importance  of  contextual  infonnation  for  segmentation  and 
stratification  and  their  focus  on  final  rather  than  interim  outcomes.  The  HMLAs  also  were  more 
likely  to  report  that  management  thought  process  performance  modeling  was  too  expensive  and 
not  essential. 
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Which,  if  any,  of  the  following  have  been  or  still  are  major  obstacles  for  the 

organization? 


1  2  3  4  5  6  7  8  9  10  11  12  IS 

U  2008  (n=145)  H  2009  (n=73) 


Figure  5-2:  2008  &  2009  —  Major  obstacles 

Key  to  Figure  5-2 

1 .  Not  enough  contextual  information  collected  for  segmentation  and  stratification 

2.  Inconsistent  measures  for  aggregation  across  the  organization 

3.  Insufficient  mentoring/coaching 

4.  Focused  on  final  rather  than  interim  outcomes 

5.  Management  thought  process  performance  modeling  was  too  expensive  and  not  essential 

6.  Resistance  to  collecting  new/additional  data  after  achieving  maturity  level  3 

7.  Too  much  time  creating  reports  instead  of  doing  thorough  analysis 

8.  Process  performance  modelers  lacking  access  to  people  with  statistical  expertise 

9.  Insufficient  alignment  of  measurement  and  analysis  with  business  goals  and  technical 
objectives 

10.  Data  not  collected  frequently  enough  to  enable  mid-course  corrections 

1 1 .  Emphasis  on  statistics  more  than  domain  knowledge  led  to  ineffective  models 

12.  Other 

13.  None  of  the  above 
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5.3  Barriers  and  Facilitators 


Finally,  we  asked  the  appraisers  about  the  extent  to  which  the  organizations  experienced  several 
common  situations  when  doing  process  performance  modeling  (see  Appendix  A,  Question  VII-2). 
Some  of  the  questions  were  stated  negatively  as  management  and  analytic  barriers  or  obstacles  to 
doing  such  work.  Others  were  stated  positively  as  management  and  analytic  facilitators  or 
enablers  of  successful  baselining  and  modeling  efforts.  The  results  may  serve  to  forewarn 
organizations  as  well  as  help  guide  and  prioritize  the  process  performance  coaching  provided  by 
the  SEI  and  others.  Responses  are  shown  in  Figure  5-3. 

On  the  positive  side,  FIMLAs  reported  that  65  percent  of  the  organizations’  managers  relied  on 
process  performance  modeling  to  keep  them  informed  about  development  status,  particularly  if 
things  started  to  go  off  track.  Also  remember  that  Figure  4-14  showed  that  about  38  percent  of 
managers  were  reported  as  understanding  process  performance  model  results  very  well  or  better. 
This  indicates  there  is  room  for  improvement:  if  managers  cannot  understand  the  results,  they  are 
less  likely  to  rely  on  them.  In  the  appraisers’  judgments,  process  perfonnance  models  were  an 
accepted  way  of  doing  business  in  nearly  half  the  cases,  with  a  third  doing  real  time  sampling  of 
process  data  when  needed.  Negative  feedback  included  organizations  having  trouble  doing 
process  performance  modeling  because  it  took  too  long  to  accumulate  enough  historical  data  (32 
percent),  building  models  without  sufficient  participation  by  management  and/or  other  important 
stakeholders  (26  percent),  and  trouble  convincing  management  about  the  value  of  process 
performance  modeling  (26  percent). 
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Following  are  a  few  statements  that  are  made  in  some  organizations  about  the  use  of  process 
performance  modeling.  Which  of  them  routinely  describe  this  organization? 
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Figure  5-3:  2009  Routine  uses  of  process  performance  models 
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6  Effects  of  Measurement  and  Analysis  on  Overall  Value  and 
Maturity  Level 


The  HMLAs’  judgments  about  the  value  of  process  performance  modeling  varied  as  a  function  of 
their  answers  to  the  survey  questions  about  analytical  methods  and  organizational  context. 
Although  there  are  some  notable  differences  between  the  two  surveys,  there  is  a  great  deal  of 
consistency  in  the  replies  of  both  groups  to  the  same  questions  and  in  the  statistical  relationships 
among  the  measures  based  on  those  questions.  These  results  are  summarized  in  Section  6.1. 

Section  6.2  shows  similar  statistical  relationships  using  achievement  of  the  organizations’  target 
maturity  level  goals  as  the  outcome  (y-variable).  As  shown  in  Section  3,  the  two  y-variables — 
achievement  of  maturity  level  goal  and  perceived  overall  value  of  process  performance  models — 
are  themselves  related  to  each  other,  so  it  should  come  as  no  surprise  that  their  respective 
relationships  with  the  various  x-variables  are  similar.  We  also  highlight  some  notable  differences 
between  the  two  y-variables  in  their  relationships  with  the  x-variables. 

6.1  Effects  on  Overall  Value 

As  in  the  2008  report,  we  used  the  question  about  the  usefulness  of  the  organizations’  process 
performance  modeling  efforts  as  the  major  indication  of  overall  value.  We  asked  HMLAs,  “How 
useful  have  process  performance  models  been  for  this  organization  overall?”  They  chose  their 
answers  from  the  following  alternatives: 

•  Extremely  valuable  -  they  rely  on  them  extensively 

•  Very  valuable  —  they  have  obtained  much  useful  information  from  them 

•  Mixed  value  -  they  have  obtained  useful  information  on  occasion 

•  Little  or  no  value 

•  Harmful,  not  helpful 

•  Don’t  know 

Questions  about  specific  performance  effects  of  process  performance  modeling  are  described  in 
Section  3  (e.g.,  about  product  quality,  fewer  project  failures,  business  growth,  and  profitability) 
(see  Appendix  A,  Question  VI-3).  These  more  specific  questions  were  not  equally  important  to  all 
of  the  surveyed  organizations.  We  asked  those  questions  first  to  provide  a  shared  context  for  the 
survey  respondents  in  making  their  judgments  about  the  overall  value  of  the  modeling.14 

6.1.1  Value  Added  by  Process  Performance  Modeling  and  Analytical  Methods 

Several  characteristics  have  collectively  come  to  be  known  as  the  “healthy  ingredients”  of  a 
CMMI-based  process  performance  model  [1]  [3]  [13]  [19]. 


14  The  same  is  so  for  the  question  about  the  use  of  the  modeling  results  for  decision  making  in  status  and  miles¬ 
tone  reviews. 
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Models  that  include  the  healthy  ingredients 

1 .  are  statistical,  probabilistic,  or  simulation  in  nature 

2.  predict  interim  and/or  final  project  outcomes 

3.  use  controllable  predictive  factors  directly  tied  to  subprocesses  or  work  activities 

4.  quantitatively  characterize  and  model  the  variation  of  the  predictive  factors  and  understand 
the  predicted  range,  uncertainty,  or  variation  of  the  outcome  performance  measures 

5.  enable  what-if  analysis  for  project  planning,  dynamic  re-planning,  and  problem  resolution 
during  project  execution 

6.  connect  upstream  activity  with  downstream  activity 

7.  enable  projects  to  achieve  mid-course  corrections  to  ensure  project  success 

As  shown  in  Section  4.1,  we  used  the  same  two  sets  of  questions  in  both  surveys  to  focus  on 
detailed  aspects  of  the  healthy  ingredients.  In  the  first  set  we  asked  about  the  aspects  of  those 
ingredients  that  the  organizations’  emphasized  in  their  models.  A  lead-in  question  asked,  “How 
much  emphasis  does  your  organization  place  on  the  following  in  its  process  perfonnance 
modeling?”  Respondents  were  asked  to  consider  the  following  six  statements  and  choose  from  the 
responses  “extensive,”  “substantial,”  “moderate,”  “limited,”  and  “little  if  any”: 

1 .  Accounting  for  uncertainty  and  variability  in  predictive  factors  and  predicted  outcomes 

2.  Factors  that  are  under  management  or  technical  control 

3.  Other  product,  contractual  or  organizational  characteristics,  resources  or  constraints 

4.  Segmenting  or  otherwise  accounting  for  uncontrollable  factors 

5.  Factors  that  are  tied  to  detailed  subprocesses 

6.  Factors  that  are  tied  to  larger,  more  broadly  defined  organizational  processes 

Their  responses  were  combined  in  a  composite  index  that  is  a  weighted  sum  of  numeric  values 
associated  with  each  response  category.15 

The  graphical  mosaic  in  Figure  6-1  shows  a  strong  association  between  the  appraisers’  perceived 
overall  value  of  the  organizations’  process  performance  modeling  efforts  and  their  judgments 
about  the  degree  to  which  that  modeling  was  consistent  with  an  emphasis  on  the  healthy 
ingredients.  Greater  emphasis  on  the  healthy  ingredients  was  more  likely  to  be  accompanied  by 
more  value  attributed  to  the  modeling  efforts,  while  less  emphasis  on  the  healthy  ingredients  was 
accompanied  by  comparably  lower  judgments  of  overall  value  added. 

Interestingly  enough,  the  appraisers  judged  the  organizations’  emphasis  on  the  healthy  ingredients 
to  be  notably  better  than  did  the  appraisal  sponsors  and  their  delegates  in  the  earlier  survey.  The 
width  of  the  mosaic  column  for  less  than  moderate  emphasis  was  considerably  wider  previously 


15  See  Appendix  C  for  further  detail  about  the  composite  measures  that  are  used  throughout  this  report.  A  very 
few  “Don’t  know”  and  “does  not  apply”  answers  were  excluded  from  the  calculations  in  this  and  the  other  com¬ 
posite  variables  that  follow  in  this  report.  However,  answers  to  the  “other”  categories  that  sometimes  exist  in  re¬ 
lated  question  sets  were  included  in  the  composite  indices. 
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and  the  right  most  two  columns  were  considerably  narrower  than  what  is  shown  here  in  Figure 
6-1.  That  is  because  the  width  of  the  mosaic  columns  corresponds  to  the  proportion  of 
respondents  who  chose  each  answer  to  the  x-variable  represented  along  the  horizontal  axis.  The 
Goodman  and  Kruskal’s  gamma  measure  of  association  with  a  value  of  .66  also  is  comparable  to 
the  value  of  .60  in  the  earlier  survey.  The  gamma  statistic  indicates  the  strength  of  the  statistical 
relationship  between  two  ordinal  categorical  variables.  In  this  example,  knowing  the  response  to 
the  question  for  emphasis  on  process  performance  models  improves  our  accuracy  of  predicting  the 
response  to  overall  value  by  66  percent. 16 


Figure  6-1:  2009  —  Relationship  between  emphasis  on  healthy  process  performance  model 

ingredients  and  overall  value  attributed  to  process  performance  models 

Similarly,  the  associations  between  overall  value  attributed  to  the  organizations’  use  of  process 
performance  modeling  and  the  individual  component  items  that  make  up  the  composite  index  of 
emphasis  on  the  healthy  ingredients  are  somewhat  stronger  in  2009  than  2008.  The  one  exception 
is  for  the  item  about  factors  tied  to  detailed  subprocesses.  Gamma  is  equal  to  .55  for  the  2008 
survey  data  but  only  .45  in  2009.  Flowever,  the  item  about  factors  tied  to  larger,  more  broadly 
defined  organizational  processes  is  more  strongly  related  to  overall  value  (gamma  =.87  in  2009  as 
contrasted  to  a  gamma  of  .59  in  2008).  Differences  in  exploratory  data  analysis  results  should  not 
be  over  interpreted  [20]  [21],  but  they  are  suggestive  nonetheless  and  can  provide  clues  for  further 
detailed  analyses. 


16  The  gamma  values  in  our  earlier  technical  report  sometimes  differ  from  what  is  shown  in  this  report  due  to  the 
way  the  statistical  software  we  used  treated  the  categorization  of  the  composite  variables.  Further  detail  about 
interpreting  the  mosaics  and  Goodman  and  Kruskal’s  gamma  also  may  be  found  in  Appendix  C. 
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That  same  pattern  exists  for  other  comparisons  between  the  two  surveys.  Still,  the  width  of  the 
columns  in  Figure  6-1  also  suggests  that  considerable  room  exists  for  improving  the  emphasis  on 
the  healthy  ingredients  of  process  performance  modeling,  which  may  in  turn  add  considerable 
business  value. 

We  asked  in  the  second  set  of  questions  about  the  extent  to  which  the  organizations’  purposes  for 
using  their  models  were  or  were  not  consonant  with  the  healthy  ingredients  of  a  CMMI-based 
process  perfonnance  model.  This  time  a  lead-in  question  asked,  “To  what  degree  are  your 
organization's  process  perfonnance  models  used  for  the  following  purposes?”  (see  Appendix  A, 
Question  IV-5).  Figure  6-2  shows  an  extremely  clear-cut  associative  relationship  between  overall 
value  and  the  five-item  composite  variable  summarizing  the  appraisers’  judgments  about  the  use 
of  process  performance  models  that  are  consistent  with  the  healthy  ingredients.  The  comparable 
gamma  value  was  .61  in  the  earlier  survey,  which  also  is  quite  high.  Moreover,  many  more 
organizations  were  judged  by  the  FIMLAs  to  be  using  their  process  performance  models  for 
purposes  consistent  with  the  healthy  ingredients  than  was  apparent  from  the  perspective  of  the 
appraisal  sponsors  in  the  previous  year’s  survey. 


Healthy  PPM 
ingredients:  Use 


Figure  6-2:  2009  —  Relationship  between  use  of  healthy  process  performance  model 

ingredients  and  overall  value  attributed  to  process  performance  models 
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The  associations  between  overall  value  and  the  individual  component  items  that  make  up  the 
composite  index  of  use  of  the  healthy  ingredients  are  shown  in  Table  1,  along  with  the  wording  of 
the  component  items.  Again,  such  differences  in  exploratory  data  analysis  should  not  be 
overemphasized,  but  note  that  the  relationships  are  consistently  stronger  in  the  HMLA  survey 
than  they  are  in  the  earlier  survey  of  representatives  from  the  appraised  organizations. 

Table  1:  Relationships  between  individual  items  about  the  of  use  of  healthy  process  performance 

model  ingredients  and  overall  value 


Strength  of  Association  (y) 

Item  wording 

2008  survey 
gammas 

2009  survey 
gammas 

Predict  final  project  outcomes 

0.57 

0.70 

Predict  interim  outcomes  during  project  execution  (e.g.,  connecting 
“upstream”  with  “downstream"  activities) 

0.47 

0.76 

Model  the  variation  of  factors  and  understand  the  predicted  range  or 
variation  of  the  predicted  outcomes 

0.50 

0.59 

Enable  “what-if”  analysis  for  project  planning,  dynamic  re-planning 
and  problem  resolution  during  project  execution 

0.62 

0.74 

Enable  projects  to  achieve  mid-course  corrections  to  ensure  project 
success 

0.59 

0.65 
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Figure  6-3  shows  a  strong  relationship  between  the  overall  value  attributed  to  process 
performance  modeling  and  the  diversity  of  models  used  to  predict  product  quality  and  project 
performance.  FIMLAs  were  asked  which  of  several  kinds  of  product  quality  and  project 
performance  outcomes  were  routinely  predicted  with  process  performance  models  in  the 
organization  about  which  they  reported.  The  composite  index  is  a  simple  sum  of  the  number  of 
component  items  checked  by  each  HMLA  surveyed.  These  include  delivered  defects,  cost  and 
schedule  duration,  accuracy  of  estimates,  type  and  severity  of  defects,  quality  of  services 
provided,  customer  satisfaction,  product  quality  attributes,  work  product  size,  and  measures  of 
ROI  or  financial  performance.  As  shown  in  the  figure,  organizations  that  maintained  and  used  a 
more  varied  suite  of  analytical  techniques  to  predict  product  quality  and  project  performance  were 
considerably  more  likely  to  find  value  in  their  modeling  than  those  that  did  not.  Although  the 
differences  are  slightly  less  consistent  than  other  comparisons  between  the  two  surveys,  a  similar 
relationship  was  seen  with  the  same  set  of  questions  in  the  earlier  survey  of  appraisal  sponsors 
and  their  delegates  (gamma  =  .56). 17 


17  Similar  results  exist  for  the  diversity  of  models  of  interim  performance  outcomes  (e.g.,  estimates  at  completion, 
escaped  defects,  and  the  other  classes  of  measures  described  in  Section  4.1).  The  relationship  of  overall  value 
with  diversity  of  models  of  interim  performance  outcomes  as  the  x  factor  is  also  strong  (y  =  .49). 
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A  more  varied  set  of  analytic  methods  also  appeared  to  pay  off  for  the  organizations  about  which 
the  HMLAs  reported,  as  it  did  in  the  2008  survey  of  appraised  high  maturity  organizations.  A 
strong  relationship  between  overall  value  attributed  to  process  performance  modeling  and  a 
composite  measure  based  on  the  mix  of  different  statistical  methods  used  by  these  organizations 
in  their  process  perfonnance  modeling  is  shown  in  Figure  6-4. ls 


Figure  6-4:  2009  —  Relationship  between  use  of  statistical  methods  and  overall  value 

attributed  to  process  perfonnance  models 

The  width  of  the  leftmost  column  in  the  mosaic  (which  indicates  more  than  a  third  of  the  cases) 
still  shows  room  for  increasing  the  use  of  appropriate  statistical  methods,  as  it  did  in  the  earlier 
survey.  The  greater  likelihood  of  finding  value  as  one  moves  to  the  right  in  the  mosaic  also 
suggests  that  improvement  in  an  organization’s  statistical  capabilities  may  be  well  worth  the 
effort. 


18  The  relationship  between  the  composite  index  and  overall  value  of  process  performance  modeling  is  essentially 
the  same  in  both  surveys  (gamma  =  .52  in  2008  as  compared  to  .53  in  2009). 
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The  associations  between  overall  value  and  the  individual  component  items  that  make  up  the 
statistical  methods  composite  index  are  shown  in  Table  2,  along  with  the  wording  of  the 
component  items.  The  relationships  are  somewhat  stronger  in  some  instances  in  the  HMLA 
survey  than  they  are  in  the  earlier  survey  of  appraised  organizations,  especially  in  the  use  of 
design  of  experiments. 


Table  2:  Relationships  between  individual  items  about  the  of  use  of  statistical  methods  for 

process  performance  modeling  and  overall  value 


Strength  of  Association  (y) 

Item  wording 

2008  survey 

2009  survey 

gammas 

gammas 

Regression  analysis  predicting  continuous  outcomes  (e.g.,  bivariate  or 
multivariate  linear  regression  or  non-linear  regression) 

0.42 

0.50 

Regression  analysis  predicting  categorical  outcomes  (e.g.,  logistic  regression 
or  loglinear  models) 

0.52 

0.44 

Analysis  of  variance  (e.g.,  ANOVA,  ANCOVA  or  MANOVA) 

0.46 

0.47 

Attribute  SPC  charts  (e.g.,  c,  u,  p,  or  np) 

0.40 

0.38 

Individual  point  SPC  charts  (e.g.,  ImR  or  XmR) 

0.48 

0.51 

Continuous  SPC  charts  (e.g.,  XbarR  or  XbarS) 

0.44 

0.32 

Design  of  experiments 

0.37 

0.63 

A  moderately  strong  relationship  also  exists  in  the  HMLA  survey  between  a  simple  summed 
composite  index  of  the  use  of  several  optimization  and  simulation  methods  for  process 
performance  modeling  and  overall  value  attributed  to  that  modeling  (see  Figure  6-5).  Recall  that 
these  include  the  following: 

•  Monte  Carlo  simulation 

•  discrete  event  simulation  for  process  modeling 

•  Markov  or  Petri-net  models 

•  probabilistic  modeling 

•  neural  networks 

•  optimization 

The  strength  of  the  relationship  is  essentially  the  same  in  both  2008  and  2009  (gamma  =  .44  and 
.45  respectively).  Fewer  than  half  of  the  respondents  selected  “none  of  the  above”  when  asked 
about  the  use  of  these  same  analytical  methods.  The  relationships  with  overall  value  were 
considerably  stronger  in  both  years  when  comparing  those  who  selected  none  of  the  above  with 
those  who  selected  one  or  more  of  the  other  options  (gamma  =  .53  and  .77  respectively).  Together 
these  results  suggest  that  more  use  of  such  analytical  methods  will  prove  to  be  worthwhile. 
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Figure  6-5:  2009  —  Relationship  between  the  number  of  optimization  methods  used  and 

overall  value  attributed  to  process  performance  models 

Moderately  strong  relationships  also  exist  for  the  use  of  automated  support  for  measurement  and 
analysis  and  the  use  of  data  quality  and  integrity  checks.  In  the  first  instance,  the  HMLAs  were 
queried  about  a  series  of  ways  that  the  organizations  provided  automated  support  for  their 
measurement-related  activities  (see  Appendix  A,  Question  III-3).  A  summary  of  the  results  is 
shown  in  Table  3. 

Table  3:  2009  —  Relationships  between  the  use  of  automated  method  to  support  measurement 
related  activities  and  overall  value  attributed  to  process  performance  modeling 


Item  wording 

Strength  of 
Association  (y) 

Data  collection  (e.g.,  online  forms  with  “tickler”  reminders,  time-stamped 
activity  logs,  static  or  dynamic  analyses  of  call  graphs  or  run-time 
behavior) 

-0.07 

Commercial  work  flow  automation  that  supports  data  collection 

0.27 

Data  management  (e.g.,  relational  or  distributed  database  packages, 
open  database  connectivity,  tools  for  data  integrity,  verification,  or 
validation) 

0.30 

Spreadsheet  add-ons  for  basic  statistical  analysis 

0.22 

Commercial  statistical  packages  that  support  more  advanced  analyses 

0.53 

Customized  spreadsheets  for  routine  analyses  (e.g.  for  defect  phase 
containment) 

0.38 

Commercial  software  for  report  preparation  (e.g.,  graphing  packages  or 
other  presentation  quality  results) 

0.52 

Composite  index  in  2009  HMLA  survey 

0.41 

As  shown  in  the  table,  the  strength  of  the  gammas  varies  widely  as  one  might  expect  in 
exploratory  analyses  with  widely  varying  numbers  of  respondents  checking  each  possible  box. 
Note  though  that  two  strong  measures  of  statistical  association  exist  for  functionality  that  supports 
more  sophisticated  analyses.  Notice  also  that  the  strength  of  the  relationships  between  each  of  the 
two  composite  indices  with  overall  value  is  essentially  the  same.  While  there  is  considerable  room 
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for  increasing  automation  in  these  organizations,  the  potential  is  there  for  it  to  pay  off  in  better 
modeling  outcomes.19 

Similar  results  can  be  seen  with  respect  to  the  use  of  data  quality  and  integrity  checks.  The 
HMLAs  were  queried  about  a  series  of  ways  that  the  organizations  performed  data  quality  and 
integrity  checks  for  their  measurement-related  activities  (see  Appendix  A,  Question  III -4).  A 
summary  of  the  results  is  shown  in  Table  4.  In  this  instance,  while  still  moderate,  the  relationship 
with  overall  value  added  by  process  performance  modeling  is  notably  stronger  in  the  HMLA 
survey  than  it  was  in  the  earlier  survey  of  appraisal  sponsors  and  their  delegates.  Once  again,  the 
table  includes  results  for  the  individual  items  from  the  HMLA  survey  only.  The  response 
alternatives  differed,  and  once  again  most  of  the  individual  item  relations  were  notably  weaker  in 
the  2008  survey.  With  our  usual  caveat  about  exploratory  data  analysis,  notice  in  the  table  that 
several  of  the  individual  item  relationships  are  quite  strong  in  the  HMLA  survey.  Only  the  items 
about  checking  for  out  of  range  and  illegal  values  (gamma  =  .49)  and  the  use  of  automated  checks 
for  consistent  data  collection  (gamma  =  .55)  were  comparably  strong  in  the  earlier  survey.  Once 
again,  there  is  considerable  room  for  improvement.  Whether  a  trend  of  increasing  recognition  of 
the  importance  of  attention  to  data  quality  and  integrity  is  emerging  in  high  maturity 
organizations — and  is  mirrored  among  certified  HMLAs — remains  to  be  seen. 

Table  4:  Relationships  between  data  quality  and  integrity  activities  and  overall  value  attributed 

to  process  performance  modeling: 


Item  wording 

Strength  of 
Association  (y) 

Check  for  out  of  range  or  other  illegal  values  in  the  recorded  data 

0.75 

Evaluate  the  number  and  distribution  of  missing  data 

0.28 

Ensure  that  missing  data  are  not  inadvertently  treated  as  zero  values 

0.62 

Check  for  precision  and  accuracy  of  the  data 

0.71 

Estimate  measurement  error  statistically 

0.35 

Check  for  inconsistent  interpretations  of  measurement  definitions 

0.28 

Check  for  consistency/reliability  of  measurement  results  and  procedures 
across  time  and  reporting  units 

0.32 

Check  for  consistency  of  classification  decisions  based  on  the  same 
information  (otherwise  known  as  inter-coder  reliability) 

0.37 

Analyze  &  address  the  reasons  for  unusual  patterns  in  the  data 
distributions,  e.g.,  outliers,  skewness,  or  other  aspects  of  non  normal 
distributions 

0.75 

Analyze  &  address  the  reasons  for  unusual  or  unanticipated 
relationships  between  two  or  more  measures 

0.25 

Automate  data  quality/integrity  checks  for  ease  of  collecting  consistent 
data 

0.61 

Composite  index  in  2008  survey  of  high  maturity  organizations 

0.36 

Composite  index  in  2009  HMLA  survey 

0.49 

19  Both  surveys  included  the  same  series  of  items,  but  the  results  are  not  directly  comparable.  The  2008  survey 
gave  the  respondents  a  series  of  ordered  response  alternatives,  the  answers  to  which  were  combined  in  a 
weighted  sum  composite  index.  The  2009  survey  used  a  series  of  checkboxes,  the  answers  to  which  were 
combined  in  a  simple  summed  index.  Hence  the  table  contains  gammas  for  the  individual  items  in  the  series  for 
the  HMLA  survey  only.  While  most  of  the  individual  item  relationships  were  notably  weaker  in  the  2008  survey, 
the  relationship  with  the  composite  index  was  essentially  the  same  in  both  surveys  (gamma  =  .42  and  .41  in 
2008  and  2009  respectively). 
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6.1.2  The  Effects  of  Organizational  Context:  Alignment  and  Coordination  of 
Measurement  and  Analysis  Activities 

The  importance  of  management  support  and  sponsorship  for  successful  process  improvement  has 
been  widely  acknowledged  in  software  and  systems  engineering  for  many  years.  Not  surprisingly, 
there  is  empirical  evidence  that  such  support  can  increase  the  likelihood  of  the  success  of 
measurement  efforts  in  general  [5]  [8]  [16].  Similar  results  exist  in  the  2008  survey  of  high 
maturity  organizations,  as  well  as  the  survey  of  HMLAs  summarized  in  this  report. 

Several  survey  questions  address  this  issue.  These  include  questions  about  various  levels  of 
management  as  well  as  other  practitioner  roles  in  the  organization.  Interestingly  enough,  the 
relationships  of  these  x-variables  with  overall  value  are  much  stronger  in  the  HMLA  survey  than 
in  the  survey  of  representatives  from  the  high  maturity  organizations  themselves. 

The  availability  of  qualified  and  well-prepared  individuals  to  do  an  organization’s  process 
performance  modeling  is  a  crucial  gauge  of  management  support.  While  the  relationship  in  the 
earlier  survey  of  high  maturity  organizations  is  only  moderately  strong  (gamma  =  .42),  the 
relationship  shown  in  Figure  6-6  is  extremely  strong  for  survey  data  of  this  kind  (gamma  =  .76). 
As  depicted  in  both  surveys  by  the  relatively  narrow  columns  on  the  left-hand  side  of  the  mosaic, 
most  of  the  survey  respondents  reported  that  the  necessary  expertise  frequently  is  available  when 
it  is  needed. 


Figure  6-6:  2009  —  Relationship  between  availability  of  well-prepared  personnel 

to  work  on  process  performance  modeling  and  overall  value  attributed  to  process 

performance  models 

We  also  asked  the  respondents  in  both  surveys  how  well  the  managers  who  use  process 
performance  model  results  understand  the  results  that  they  use  (see  Appendix  A,  Question  II-4). 
The  relationship  between  the  FIMLAs’  answers  to  that  question  and  what  they  tell  us  elsewhere  in 
the  questionnaire  about  the  overall  value  of  process  performance  modeling  is  shown  in  Figure  6-7. 
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Once  again,  the  HMLAs’  answers  to  the  two  questions  co-vary  quite  consistently  with  each  other. 
While  the  comparable  relationship  in  the  earlier  survey  also  is  quite  strong  (gamma  =  .59),  the 
statistical  relationship  in  the  HMLA  survey  is  an  extremely  strong  one. 


Extremely  Valuable 


Very  Valuable 


Mixed  Value 


Little  Value 


■ 


y  =  .83,  n  =  72 


Figure  6-  7:  2009  —  Relationship  between  managers  ’  understanding  of  process 

performance  model  results  and  overall  value  attributed  to  process  performance  models 

As  shown  in  Table  5,  similarly  strong  relationships  exist  with  a  series  of  questions  about  how  well 
those  who  create  the  organizations’  process  performance  models  and  baselines  understand  the 
intent  of  CMMI  (see  Appendix  A,  Question  II-3).  Once  again  the  relationships  are  considerably 
stronger  in  the  HMLA  survey. 

Table  5:  Relationships  between  modelers  ’  understanding  of  the  intent  of  CMMI  process 

performance  baselines  and  models  and  overall  value  attributed  to  the  model  outcomes 


Strength  of  Association  (y) 

Item  wording 

2008  survey 
gammas 

2009  survey 
gammas 

The  CMMI  definition  of  a  process  performance  baseline 

0.27 

0.72 

The  CMMI  definition  of  a  process  performance  model 

0.41 

0.78 

The  circumstances  when  process  performance 
baselines  are  useful 

0.45 

0.76 

The  circumstances  when  process  performance  models 
are  useful 

0.47 

0.71 

To  quote  from  our  earlier  survey  of  high  maturity  organizations,  “The  importance  of  stakeholder 
involvement  is  widely  acknowledged  by  measurement  experts.  The  inclusion  of  key  stakeholders 
in  deciding  what  to  measure  and  why  to  do  so  is  a  basic  notion  of  goal-driven  measurement  [14] 
[15].  It  also  is  crucial  for  the  CMMI  Measurement  and  Analysis  process  area,  particularly  in 
specific  goal  1  and  specific  practice  1.1  which  are  meant  to  ensure  that  measurement  objectives 
and  activities  are  aligned  with  the  organizational  unit’s  information  needs  and  objectives,  and  in 
specific  practice  2.4  which  emphasizes  the  importance  of  reporting  the  results  to  all  relevant 
stakeholders.”  Similar  to  the  influence  of  management  support,  empirical  evidence  exists  showing 
that  key  stakeholder  involvement  can  increase  the  likelihood  of  the  success  of  measurement 
programs  in  general  [5]  [8]  [16].  We  also  continue  to  see  incidents  where  insufficient  stakeholder 
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involvement  in  the  creation  of  process  performance  baselines  and  models  has  seriously 
jeopardized  their  productive  use. 

As  shown  in  Figure  6-8,  an  extremely  strong  relationship  exists  between  our  composite  measure 
of  stakeholder  involvement  in  setting  goals  and  deciding  on  plans  of  action  for  measurement  and 
analysis  and  the  FIMLA  judgments  of  the  overall  value  of  using  process  performance  models  in 
the  organizations  about  which  they  reported.  Once  again  the  comparable  relationship  was  only 
moderately  strong  in  the  earlier  survey.  The  apparent  effects  of  better  alignment  of  the  models 
with  stakeholder  input  were  seen  much  more  clearly  there  as  mediated  by  other  interim  factors.20 


Figure  6-8:  Relationship  between  stakeholder  involvement  and  overall  value 

attributed  to  process  performance  models 


20  This  can  be  seen  in  Figures  52  through  54  in  the  report  based  on  the  2008  survey  of  high  maturity  organiza¬ 
tions. 
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Further  detail  about  the  relationships  for  the  individual  stakeholder  roles  in  the  2009  survey  is 
shown  in  Table  6.  Only  two  associative  relationships  in  the  earlier  survey  are  moderately  strong: 
gamma  =  .43  for  the  executive  and  senior  managers,  and  gamma  =  .44  for  the  program  or  product 
line  middle  managers. 

Table  6:  Relationship  between  involvement  by  individual  stakeholder  roles  and  overall  value 

attributed  to  process  performance  models 


Stakeholder  role 

Strength  of 
Association  (y) 

Customers 

0.38 

Executive  and  senior  managers 

0.59 

Middle  managers  (e.g.,  program  or  product  line) 

0.66 

Project  managers 

0.59 

Project  engineers  and  other  technical  staff 

0.33 

Process  and  quality  engineers 

0.35 

Measurement  specialists 

0.49 
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We  also  asked  the  respondents  in  both  surveys  to  characterize  the  measurement-related  training 
that  was  available  in  the  organizations  about  which  they  reported  (see  Appendix  A,  Question  II- 
1 ).  Comparable  results  are  not  available  from  the  earlier  survey  of  appraised  organizations  since 
the  measures  used  then  were  based  largely  on  the  duration  of  the  training.21  Notice  however  in 
Figure  6-9  that  there  is  a  very  strong  associative  relationship  between  the  quality  of  training  for 
project  managers  reported  by  the  appraisers  and  their  judgments  about  the  overall  value  of  process 
performance  modeling  in  those  same  organizations.  Notice  also  by  the  width  of  the  columns  in  the 
mosaic  that  the  majority  of  the  HMLAs  characterize  the  training  as  good  or  excellent. 


Figure  6-9:  Relationship  between  quality  of  training  available  for  project 

managers  and  overall  value  attributed  to  process  performance  models 


21  See  Section  6.5  of  the  earlier  report.  Composite  measures  were  calculated  separately  for  managers  and  those 
responsible  for  doing  the  process  performance  modeling.  The  relationship  for  the  managers  was  moderately 
strong  at  best  (gamma  =  .30).  Training  for  the  modelers  appeared  to  have  equally  little  direct  effect  (gamma  = 
.29),  and  no  indirect  effects  were  noticeable  in  the  2008  survey  data. 
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As  shown  in  Table  7,  similarly  strong  relationships  are  apparent  for  others  from  the  perspectives 
of  the  appraisers. 

Table  7:  Relationship  between  quality  of  training  available  for  different  practitioner  roles  and 
overall  value  attributed  to  process  performance  models 


Practitioner  role 

Strength  of 
Association  (y) 

Executive  and  senior  managers 

0.66 

Middle  managers  (e.g.,  program  or  product  line) 

0.56 

Project  managers 

0.66 

Project  engineers  and  other  technical  staff 

0.59 

Process  or  quality  engineers 

0.76 

Process  performance  model  builders  &  maintainers  (e.g.,  Six  Sigma 
black  belts  or  other  measurement  specialists) 

0.69 

Coaches  &  mentors  who  assist  the  model  builders  and  maintainers  (e.g., 
Six  Sigma  master  black  belts) 

0.54 

Those  who  collect  and  manage  the  baseline  data  (e.g.,  Six  Sigma  green 
belts,  other  project  engineers  or  EPG  members) 

0.77 

Users  of  the  models 

0.56 

We  added  a  related  question  to  the  survey  of  certified  HMLAs  in  2009.  We  asked  the  HMLAs  to 
describe  each  “organization’s  documentation  of  its  process  performance  and  quality  measurement 
results,  (e.g.,  in  the  organization's  measurement  repository  or  process  asset  library).”  The  response 
categories  included  “excellent,”  “good,”  “adequate,”  “fair,”  “poor,”  and  “Don’t  know”  (see 
Appendix  A,  Question  VI-2).  An  extremely  strong  relationship  between  the  HMLAs’  answers  to 
that  question  and  their  judgments’  about  the  overall  value  of  the  organizations’  process 
performance  modeling  is  shown  in  Figure  6-10. 


Figure  6-10:  2009  —  Documentation  of  process  performance  and  quality  measurement  results 
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Another  extremely  strong  statistical  relationship  is  shown  in  Figure  6-11.  The  perceived  overall 
value  attributed  to  process  performance  modeling  varied  in  the  FIMLA  survey  with  the  extent  to 
which  the  model  results  were  used  to  inform  decision  making  in  status  and  milestone  reviews, 
even  more  than  it  did  in  the  earlier  survey  of  appraised  organizations.  The  strength  of  association 
reported  for  the  earlier  survey  was  very  strong  for  survey  data  of  this  kind  in  software  and  systems 
engineering  (gamma  =  .67).  As  shown  in  Figure  6-11,  it  has  risen  to  a  value  of  .88  in  the  more 
recent  survey.  The  extent  to  which  the  model  results  were  used  is  a  valuable  y-variable  in  its  own 
right.  It  undoubtedly  is  a  function  of  management  support  and  the  pertinence  of  the  model  results. 
As  we  said  earlier  in  Section  1.4  and  reiterate  in  Appendix  C,  the  extent  to  which  process 
performance  model  predictions  are  used  to  inform  decisions  also  is  a  function  of  the  quality  of 

the  models  themselves  and  the  analytical  methods  used  to  build  them.  Cause  and  effect  is 

22 

probably  recursively  reciprocal  over  time. 


Figure  6-11:  Relationship  between  use  of  process  performance  model  predictions  in  status 

and  milestone  reviews  and  overall  value  attributed  to  process  performance  models 


22  One  can  argue  that  organizations  use  the  models  in  their  reviews  because  they  find  value  in  them  rather  than 
the  other  way  around.  The  causal  direction  undoubtedly  is  reciprocal  over  time.  The  same  is  true  for  other  x  va¬ 
riables  as  well.  Yet,  as  can  be  seen  in  Section  6.6  of  [1],  some  of  those  who  perceive  more  overall  value  from 
their  modeling  activities  also  are  consistently  more  likely  to  use  them  in  their  reviews. 


61  |  CMU/SEI-201 0-TR-022 


6.1.3  Other  Effects  of  Challenges,  Barriers,  and  Facilitators  of  Successful 
Measurement  and  Analysis 

In  Section  5,  we  listed  three  series  of  questions  about  managerial  and  technical  challenges  that  are 
often  cited  as  barriers  and  facilitators  of  successful  measurement  and  analysis  activities.  The  first 
series  asked  the  survey  respondents  to  characterize  the  various  kinds  of  technical  challenges  that 
projects  sometimes  face  (see  Appendix  A,  Question  VI- 1).  Once  again,  the  questions  were 
phrased  somewhat  differently  in  the  two  surveys,  so  the  results  are  not  directly  comparable. ' 
Nevertheless  the  results  are  of  interest.24 

Table  8  shows  the  relationships  between  (1)  whether  or  not  the  HMLAs  said  that  each  statement 
typically  described  the  projects  and  (2)  their  judgments  about  the  overall  value  to  the  organiza¬ 
tions  of  their  respective  process  performance  modeling  activities.  As  shown  in  the  Table  8,  the 
relationships  between  the  individual  items  that  refer  to  technical  challenge  about  system  and 
product  characteristics  per  se  are  weakly  related  to  overall  value  at  best. 

Table  8:  Relationship  between  technical  challenges  that  projects  sometimes  face 


and  overall  value  attributed  to  process  performance  models 


Technical  challenge 

Strength  of 
Association  (y) 

Initial  project  requirements  are  not  well  defined 

-0.25 

Requirements  change  significantly  throughout  the  life  of  the  projects 

0.18 

There  is  little  or  no  precedent  for  the  kind  of  work  they  are  doing 

-0.29 

Significant  constraints  are  placed  on  product  quality  attributes  (e.g. 
reliability,  scalability,  security,  supportability,  etc.) 

0.30 

The  size  of  the  development  effort  is  large 

0.36 

The  technology  needed  for  the  projects  is  not  mature 

-0.19 

There  are  extensive  needs  for  interoperability  with  other  systems 

-0.16 

Insufficient  resources  (e.g.  people,  funding)  are  available  to  support  the 
projects 

-0.59 

Insufficient  skills  and  subject  matter  expertise  are  available  to  support 
the  projects 

-0.48 

These  are  much  like  the  results  in  the  earlier  survey  of  high  maturity  organizations.  Unlike  studies 
based  on  projects  from  organizations  that  exhibit  a  wider  range  of  process  maturity  [17]  [18],  high 
maturity  organizations  may  be  better  able  to  handle  difficult  projects.  However,  notice  the  last 
two  items  in  the  list,  where  the  relationships  are  quite  strong.  They  refer  to  issues  of  management 
support  for  the  work 

The  second  set  of  questions  asked  whether  or  not  a  series  of  statements  concerning  measurement 
and  analysis  activities  described  circumstances  that  “have  been  or  still  are  major  obstacles  for  the 
organization”  (see  Appendix  A,  Question  VII- 1).  Most  of  the  obstacles  make  specific  reference  to 
process  perfonnance  modeling,  and  the  first  five  are  worded  to  address  aspects  of  the  healthy 


23  The  earlier  survey  questions  asked  the  appraisal  sponsors  and  their  delegates  to  characterize  the  extent  to 
which  each  statement  characterized  projects  in  their  respective  organizations.  However,  to  reduce  the  length  of 
the  survey  and  its  burden  on  their  time,  the  HMLAs  were  simply  asked  which  ones  typically  describe  the  organi¬ 
zations  about  which  they  reported. 

24  See  Section  6.6  in  the  report  that  summarizes  the  results  of  the  2008  survey  of  high  maturity  organizations  in 
much  greater  detail. 
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ingredients  of  process  performance  models.  Four  of  them  refer  to  measurement  and  analysis  more 
generally,  but  they  too  are  closely  coupled  to  process  performance  modeling  activities.25  The 
extent  to  which  the  respondents’  answers  concerning  obstacles  to  measurement  and  analysis 
activities  are  associated  with  their  replies  to  the  question  about  the  overall  value  of  process 
performance  modeling  in  both  surveys  is  summarized  in  Table  9. 

Note  that  none  of  the  associative  relationships  between  overall  value  and  the  individual  obstacles 
is  more  than  moderately  strong  in  the  2008  survey  of  high  maturity  organizations  (gammas  <.40). 
Whether  or  not  the  sponsors  and  their  delegates  recognize  the  existence  of  these  obstacles  tells  us 
relatively  little  about  their  judgments  about  the  overall  value  of  process  performance  modeling  in 
their  organizations.  In  contrast,  in  only  one  instance  is  one  of  the  same  obstacles  recognized  by 
the  HMLAs  in  2009  less  than  moderately  strongly  related  to  the  HMLAs’  judgments  about  overall 
value.  Most  of  the  other  relationships  from  the  HMLA  survey  are  quite  strong  indeed.  The 
relationships  between  overall  value  as  judged  by  the  HMLAs  and  their  judgments  about  whether 
or  not  measurement  and  analysis  practices  are  aligned  with  the  organizations’  business  and 
technical  goals  and  objectives  (gamma  =  .83)  and  whether  or  not  sufficient  mentoring  and 
coaching  is  available  for  the  process  performance  model  developers  (gamma  =  .76)  are  extremely 
strong. 

While  the  relationship  between  whether  or  not  the  sponsors  and  their  delegates  checked  “none  of 
the  above”  and  their  judgments  about  the  overall  value  of  process  performance  modeling  in  their 
organizations  is  quite  strong  (gamma  =  .53),  the  comparable  relationship  from  the  HMLA  survey 
is  extremely  strong  (gamma  =  .78). 


Table  9:  Relationships  between  obstacles  to  measurement  and  analysis  and  overall  value  of 

process  performance  model  outcomes 


Strength  of  Association  (y) 

Obstacles  faced  by  organizations 

2008  survey 

2009  survey 

Focused  only  on  final  project  outcomes  rather  than  interim  outcomes 

-0.23 

-0.50 

Didn't  collect  data  frequently  enough  to  help  projects  make  mid-course 
corrections 

-0.29 

-0.59 

Failed  to  collect  enough  contextual  information  for  proper  segmentation 
and  stratification 

-0.34 

-0.57 

Failed  to  achieve  enough  consistency  in  their/our  measures  to  aggregate 
and  disaggregate  them  properly  across  the  organization 

-0.16 

-0.66 

Failed  to  sufficiently  align  and  prioritize  their/our  measurement  and 
analysis  practices  with  their/our  business  and  technical  goals  and 
objectives 

-0.38 

-0.83 

Encountered  resistance  to  collecting  new  or  additional  data  after 
achieving  maturity  level  3 

-0.31 

-0.60 

Management  thought  that  process  performance  modeling  would  be  an 
expensive  overhead  function  rather  than  an  essential  part  of  project  work 

-0.36 

-0.50 

Spent  too  much  time  creating  reports  for  management  review  instead  of 
doing  thorough  analysis 

-0.02 

-0.51 

Emphasized  statistics  more  than  domain  knowledge  and  ended  up  with 
ineffective  models 

-0.25 

-0.65 

Didn't  provide  sufficient  mentoring  and  coaching  for  the  individuals 
responsible  for  developing  the  models 

-0.15 

-0.76 

25  Obstacles  5  through  7  in  Table  9  are  worded  generally  but  are  equally  pertinent  for  process  performance  modeling. 
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Process  performance  modelers  don't  have  sufficient  access  to  people 
with  statistical  expertise 

-0.36 

-0.67 

None  of  the  above 

0.53 

0.78 

Recognizing  that  the  strength  of  the  statistical  relationships  tends  to  be  weaker  in  2008,  the 
consistency  of  the  differences  shown  in  Table  9  is  highly  unlikely  by  chance  alone.  Why  these 
differences  exist  is  not  immediately  clear.  Perhaps  the  perspectives  of  the  sponsors  or  the 
appraisers  are  more  accurate,  or  there  may  be  a  trend  developing  over  time.  Regardless  of  the 
reasons,  these  results  are  worth  further  consideration  by  those  who  wish  to  enhance  the  value  for 
their  organizations  of  state  of  the  art  measurement  and  analysis  activities. 

The  third  set  of  questions  asked  about  another  set  of  statements  that  may  or  may  not  have  been 
made  in  the  organization  about  the  use  of  process  perfonnance  modeling  (see  Appendix  A,  Ques¬ 
tion  VII-2).  These  questions  also  were  phrased  differently  in  the  two  surveys.  The  appraisal  spon¬ 
sors  and  their  delegates  in  the  earlier  survey  were  asked  to  choose  from  an  ordinal  range  of  possi¬ 
ble  answers  about  how  well  each  statement  described  their  organizations  (ranging  from  “almost 
entirely”  to  “hardly  at  all”).  To  reduce  their  time  burden,  the  HMLAs  were  asked  only  to  select  as 
many  as  applied  to  the  organizations  about  which  they  reported.  Although  the  results  are  not  di¬ 
rectly  comparable,  they  are  somewhat  consistent  across  the  two  surveys.  Table  10  shows  the  rela¬ 
tionships  between  the  survey  respondents’  answers  to  the  each  of  the  questions  about  common 
barriers  and  facilitators  in  the  use  of  process  performance  modeling  and  their  judgments  about  the 
overall  value  added  by  that  modeling.  The  statements  in  the  table  are  shown  in  two  lists  corres¬ 
ponding  to  two  composite  measures  that  were  used  in  our  report  describing  the  results  of  the  2008 
survey.  Some  of  the  questions  are  stated  negatively  as  barriers  while  others  are  stated  positively  as 
facilitators.  The  negatively  stated  items  were  reverse  scored  when  combined  in  the  two  composite 
measures  for  both  surveys;  however,  for  simplicity’s  sake,  the  gamma  coefficients  in  the  table 
show  the  strength  of  the  relationships. 
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Table  10:  Relationships  between  barriers  and  facilitators  of  process  performance  modeling  and 

overall  value  of  the  model  outcomes 


Strength  of  Association  (y) 

Facilitators  for  modeling 

2008  survey 

2009  survey 

Doing  process  performance  modeling  has  become  an  accepted 
way  of  doing  business  there/here 

0.69 

0.89 

They/we  thought  they/we  knew  what  was  driving  process 
performance,  but  process  performance  modeling  has  taught 
them/us  otherwise 

0.32 

0.28 

Managers  want  to  know  when  things  are  off-track 

0.36 

0.69 

Managers  are  unwilling  to  fund  new  work  when  the  outcome  is 
uncertain 

0.21 

0.06 

Use  data  mining  when  similar  but  not  identical  electronic  records 
exist 

0.11 

0.12 

Do  real  time  sampling  of  current  processes  when  historical  data 
are  not  available 

0.12 

0.68 

They/we  create  their/our  baselines  from  paper  records  for 
previously  unmeasured  attributes 

0.23 

-0.8026 

Barriers  to  modeling 

Have  trouble  doing  process  performance  modeling  because  it 
takes  too  long  to  accumulate  enough  historical  data 

-0.58 

-0.49 

Made  decisions  about  the  models  built  without  sufficient 
participation  by  management  or  other  important  stakeholders 

-0.41 

-0.64 

Had  trouble  convincing  management  about  value  of  doing 
process  performance  modeling 

-0.51 

-0.72 

The  messenger  has  been  shot  for  delivering  bad  news  based  on 
process  performance  model  predictions 

-0.20 

-0.55 

Composite  measures 

All  negatively  stated  items 

-0.40 

-0.71 

All  positively  stated  items 

0.38 

0.70 

As  is  typical  for  comparisons  across  the  two  surveys,  all  but  three  of  the  relationships  are  in  fact 
stronger  in  the  HMLA  survey.  Notice  however  that  some  of  the  relationships  from  the  survey  of 
high  maturity  organizations  are  at  least  moderately  strong,  and  two  are  very  strong.  More 
importantly,  the  composite  measures  from  the  survey  of  high  maturity  organizations  are  quite 
strong  as  well.  Not  all  organizations  can  be  expected  to  experience  the  same  barriers  and 
facilitators.  The  composite  measures  from  the  earlier  survey  are  weighted  summed  indices,  which 
may  be  more  indicative  of  the  underlying  relationships  than  any  single  item  response.  Note  also 
that  the  last  three  facilitators  (listed  under  exemplary  modeling  approaches  in  the  earlier  survey) 
remained  relatively  uncommon  in  2008.  The  HMLAs  by  2009  may  have  been  more  likely  to 
recognize  the  importance  of  such  approaches  than  practitioners  and  appraisers  in  even  the  recent 
past. 

6.2  Effects  on  Achieved  Maturity  Level:  How  Overall  Value  Relates  to  Achievement  of 
Maturity  Level  Targets 

As  shown  in  Section  6.1,  the  ways  in  which  process  performance  modeling  and  various  analytical 
methods  are  used  vary  concomitantly  and  consistently  with  the  HMLAs’  judgments  about  how 


26  We  consider  this  result  to  be  a  statistical  artifact  since  only  three  cases  responded  yes. 
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useful  the  model  results  have  been  for  the  organizations  about  which  they  reported.  The  same  is 
true  for  several  factors  related  to  management  support  and  the  alignment  and  coordination  of  the 
organizations’  measurement  and  analysis  activities.  But  how  are  those  same  factors  related  to  the 
achievement  of  the  organizations’  high  maturity  level  goals  for  their  appraisals? 


There  is  in  fact  a  very  strong  relationship  between  the  HMLAs’  judgments  about  the  usefulness  of 
process  perfonnance  models  and  whether  or  not  the  organizations  achieved  their  target  maturity 
level  goals.  As  already  shown  in  Figure  3-2,  Figure  6-12  repeats  that  relationship  graphically  in  a 
paired  bar  chart  (gamma  =  .70). 27  Several  of  the  same  x-variables  that  are  closely  related  to 
overall  value  also  are  related  to  whether  or  not  the  organizations  reached  their  appraised  maturity 
level  goals.  The  rest  of  this  section  describes  the  strongest  relationships  with  achievement  of  the 
maturity  level  targets  and  highlights  those  where  the  strength  of  the  relationships  differ  when 
overall  value  is  used  as  the  dependent  variable  (y). 


2009 
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Figure  6-12:  2009  —  Relationship  between  achievement  of  appraised  high  maturity  level 

targets  and  overall  value  attributed  to  process  performance  models 


27  Paired  bar  charts  can  illustrate  some  of  the  same  information  as  the  mosaics  shown  in  Section  6.1 ;  however, 
such  bar  charts  can  become  quite  cluttered  when  the  variables  have  more  than  two  categories  each. 
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6.2.1  Value  Added  by  Process  Performance  Modeling  and  Analytical  Methods 

Figure  6-13  shows  that  an  emphasis  on  the  healthy  ingredients  of  process  performance  models  is 
closely  related  to  achievement  of  the  organizations’  high  maturity  level  goals.  The  Mann- Whitney 
test  shows  that  the  two  groups  are  distinguishable  (p  <002).  As  a  matter  of  fact,  this  relationship 
is  somewhat  stronger  than  the  comparable  relationship  between  the  emphasis  on  healthy 
ingredients  and  the  HMLAs’  judgments  about  the  overall  value  added  by  the  organizations’ 
process  perfonnance  modeling  (gamma  =  .66).  Although  the  majority  of  the  individual 
subquestions  tend  to  be  more  strongly  related  to  overall  value,  consistency  of  replies  as 
summarized  in  the  weighted  summed  index  is  more  strongly  related  to  achievement  of  the 
appraisal  goals. 


70% 


<Moderate  Moderateto  Toward  but  Substantialor 
<midway  toward  substantial  better_ 

substantial 


I  Did  Not 
Achieve 
MLGoal 
(n=21) 

I  Achieved 
MLGoal 
(n=44) 


y  =  .75,  n  =  65 


Figure  6-13:  2009  —  Relationship  between  emphasis  on  healthy  process  performance  model 

ingredients  and  achievement  of  appraised  high  maturity  level  target 
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That  said,  it  is  worth  thinking  about  the  importance  of  the  individual  items.  Recognize  that  there 
is  variation  among  the  survey  respondents  in  how  they  interpret  the  intended  meaning  of  the 
individual  subquestions,  and  that  can  account  for  statistical  noise  that  may  affect  the  gamma 
values.  Nevertheless,  with  the  exception  of  the  item  about  factors  tied  to  more  broadly  defined 
organizational  processes,  the  magnitude  of  the  relationships  with  both  y-variables  are  reasonably 
similar.  All  of  the  relationships  with  the  individual  items  in  Table  1 1  are  strongly  related  to 
achievement  of  the  organizations’  appraisal  target  goals. 

Table  11:  Comparison  of  relationships  between  emphasis  on  healthy  process  performance  model 
ingredients  with  achievement  of  target  goals  and  overall  value  of  process  performance 

model  outcomes 


Strength  of  Association  (y) 

Item  wording 

Achievement 
of  appraisal 
target  goals 

Overall  value 
added  by 
process 
performance 
models 

Accounting  for  uncertainty  and  variability  in  predictive 
factors  and  predicted  outcomes 

0.50 

0.68 

Factors  that  are  under  management  or  technical  control 

0.56 

0.67 

Other  product,  contractual  or  organizational 
characteristics,  resources  or  constraints 

0.61 

0.53 

Segmenting  or  otherwise  accounting  for  uncontrollable 
factors 

0.46 

0.59 

Factors  that  are  tied  to  detailed  subprocesses 

0.51 

0.45 

Factors  that  are  tied  to  larger,  more  broadly  defined 
organizational  processes 

0.49 

0.87 

Weighted  summed  index  in  2009  FIMLA  survey 

0.75 

0.66 
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The  comparable  relationship  between  the  use  of  the  healthy  process  performance  model 
ingredients  with  achievement  of  the  organizations’  appraisal  target  goals  is  extremely  strong 
(Figure  6-14).  Again,  the  Mann- Whitney  test  confirms  the  distinction  between  the  two  groups  (p 
<.0001).  Unlike  emphasis  on  the  healthy  ingredients  however,  the  relationships  with  use  of  the 
healthy  ingredients  are  even  more  closely  associated  with  achievement  of  the  organizations’  target 
maturity  level  goals  than  they  are  with  the  HMLA’s  judgments  about  how  useful  process 
performance  modeling  has  been  for  the  organizations  overall  (see  Table  12).  Note  especially  the 
item  about  the  achievement  of  mid-course  corrections  to  ensure  project  success. 
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Figure  6-14:  2009  —  Relationship  between  use  of  healthy  process  performance  model 

ingredients  and  achievement  of  appraised  high  maturity  level  target 


Table  12:  Comparison  of  relationships  between  use  of  healthy  process  performance  model 

ingredients  with  achievement  of  target  goals  and  overall  value  of  process  performance 

model  outcomes 


Strength  of  Association  (y) 

Item  wording 

Achievement 
of  appraisal 
target  goals 

Overall  value 
added  by 
process 
performance 
models 

Predict  final  project  outcomes 

0.75 

0.70 

Predict  interim  outcomes  during  project  execution  (e.g., 
connecting  “upstream”  with  “downstream”  activities) 

0.83 

0.76 

Model  the  variation  of  factors  and  understand  the 
predicted  range  or  variation  of  the  predicted  outcomes 

0.72 

0.59 

Enable  “what-if”  analysis  for  project  planning,  dynamic 
re-planning  and  problem  resolution  during  project 
execution 

0.75 

0.74 

Enable  projects  to  achieve  mid-course  corrections  to 
ensure  project  success 

0.90 

0.65 

Weighted  summed  index  in  2009  HMLA  survey 

0.84 

0.82 
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As  is  true  with  respect  to  the  HMLA’s  judgments  about  how  useful  process  performance 
modeling  has  been  for  the  organizations  overall,  the  use  of  several  statistical  methods  also  is 
closely  associated  with  whether  or  not  the  organizations  reached  their  target  maturity  level  goals. 
Figure  6-15  summarizes  the  relationship  between  the  weighted  summed  index  of  the  use  of 
statistical  methods  and  achievement  of  their  high  maturity  level  goals. 
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Figure  6-15: 


2009  -  Relationship  between  use  of  statistical  methods  and  achievement  of 
appraised  high  maturity  level  targets 
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A  majority  of  the  organizations  that  did  not  achieve  their  maturity  level  goals  reported  less  than 
consistently  moderate  use  of  the  methods  listed  in  Table  13  (see  Appendix  A,  Question  V-l). 
None  of  them  reported  use  of  the  methods  that  was  consistently  substantial  or  better.  In  contrast, 
almost  half  of  the  organizations  that  did  achieve  their  maturity  level  goals  were  likely  to  have 
used  those  statistical  methods  more  often  in  their  process  performance  models. 

Recognizing  that  the  survey  questions  do  not  distinguish  how  well  or  poorly  the  methods  were 
used  but  only  their  frequency  of  use,  the  weighted  summed  index  is  reasonably  similarly  related 
to  both  y- variables.  However,  as  shown  in  the  table,  three  relationships  are  similar  with  both  y- 
variables  and  four  are  not.  The  number  of  organizations  for  which  the  HMLAs  reported 
substantial  and  extensive  use  is  much  lower  for  design  of  experiments  and  categorical  regression. 

Table  13:  Comparison  of  relationships  between  use  of  statistical  methods  with  achievement  of 
target  goals  and  overall  value  of  process  performance  model  outcomes 


Strength  of  Association  (y) 

Statistical  method 

Achievement 
of  appraisal 
target  goals 

Overall  value 
added  by 
process 
performance 
models 

Regression  analysis  predicting  continuous  outcomes 
(e.g.,  bivariate  or  multivariate  linear  regression  or  non¬ 
linear  regression) 

0.51 

0.50 

Regression  analysis  predicting  categorical  outcomes 
(e.g.,  logistic  regression  or  loglinear  models) 

0.39 

0.44 

Analysis  of  variance  (e.g.,  ANOVA,  ANCOVA  or 
MANOVA) 

0.58 

0.47 

Attribute  SPC  charts  (e.g.,  c,  u,  p,  or  np) 

0.58 

0.38 

Individual  point  SPC  charts  (e.g.,  ImR  or  XmR) 

0.51 

0.51 

Continuous  SPC  charts  (e.g.,  XbarR  or  XbarS) 

0.58 

0.32 

Design  of  experiments 

0.50 

0.63 

Weighted  summed  index  in  2009  HMLA  survey 

0.58 

0.53 

Perhaps  the  most  important  differences  in  relationships  between  the  two  y-variables  are  those  with 
respect  to  the  use  of  simulation  and  other  optimization  methods.  Remember  that  we  asked 
specifically  about  Monte  Carlo  simulation,  discrete  event  simulation  for  process  modeling, 
Markov  or  Petri-net  models,  probabilistic  modeling,  neural  network,  optimization,  and  the 
residual  category  other  (see  Appendix  A,  Question  V-3). 
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As  shown  in  Figure  6-16  an  extremely  strong  relationship  is  apparent  between  a  simple  summed 
index  of  the  use  of  these  analytical  methods  and  achievement  of  the  organizations’  appraised  high 
maturity  level  targets  (p  <  .0000  for  the  Mann- Whitney  test). 


50% 

45% 

40% 

35% 

30% 

25% 

20% 

15% 

10% 

5% 

0% 


Number  of  Simulation/Optimization  Techniques  Used 


Did  Not 
Achieve 
ML  Goal 
(n=19) 

Did 

Achieve 
ML  Goal 
(n=42) 


2  3 _ 

y  =  .89,  n  =  61 


Figure  6-16:  2009  —  Relationship  between  number  of  simulation/optimization  techniques  used 

and  achievement  of  high  maturity  goal 
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Figure  6-17  summarizes  the  same  relationship  for  each  of  the  individual  methods,  including  “none 
of  the  above”  as  a  choice.28  Clearly  the  organizations  that  these  FIMLAs  reported  had  achieved 
their  target  maturity  levels  were  more  likely  to  have  used  simulation  and  other  optimization 
methods  for  their  process  perfonnance  modeling  than  were  those  that  did  not  achieve  their 
appraisal  targets. 
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2009  —  Relationship  between  use  of  simulation  and  other  optimization  methods 
and  achievement  of  appraised  high  maturity  level  targets 


28  None  of  the  HMLAs  reported  having  seen  the  use  of  neural  networks  in  the  appraised  organizations. 
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The  associative  relationships  in  Table  14  are  consistently  stronger  with  achievement  of  the 
organizations’  appraisal  target  goals  than  they  are  with  the  appraisers’  judgments  of  overall  value 
added  by  the  use  of  process  performance  models.  However,  these  differences  may  be  because 
such  analytical  methods  are  still  relatively  uncommon  in  high  maturity  organizations.  Note  also  in 
the  table  that  the  relationships  with  both  y-variables  are  extremely  strong  when  the  HMLAs 
selected  “none  of  the  above”  for  the  use  of  these  particular  methods.  Organizations  that  did  not 
use  any  of  these  methods  were  much  less  likely  to  achieve  their  maturity  level  targets.  These 
results  suggest  that  increased  use  of  simulation  and  other  still  less  common  optimization  methods 
will  prove  to  be  quite  valuable  as  they  are  used  more  widely. 

Table  14:  Comparison  of  relationships  between  use  of  simulation  and  other  optimization  methods 
with  achievement  of  target  goals  and  overall  value  of  process  performance  model 

outcomes 


Optimization  method 

Strength  of  Association  (y) 

Achievement 
of  appraisal 
target  goals 

Overall  value 
added  by 
process 
performance 
models 

Monte  Carlo  simulation 

0.51 

0.24 

Discrete  event  simulation  for  process  modeling 

0.80 

0.55 

Markov  or  Petri-net  models 

1.00 

0.61 

Probabilistic  modeling 

0.91 

0.33 

Neural  networks 

* 

* 

Optimization 

0.50 

0.04 

None  of  the  above 

-0.87 

-0.77 

Weighted  summed  index  in  2009  HMLA  survey 

0.89 

0.45 
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6.2.2  The  Effects  of  Organizational  Context:  Alignment  and  Coordination  of 
Measurement  and  Analysis  Activities 


Documentation  of  process  performance  and  quality  measurement  results  provides  another 
indicator  of  the  recognition  of  value  for  analytical  activities,  as  mentioned  briefly  in  Section  4.3. 
Figure  6-18  shows  that  the  appraisers’  judgments  of  the  quality  of  the  organizations’ 
documentation  of  their  process  performance  and  quality  measurement  results  is  very  strongly 
related  to  achievement  of  the  organizations’  appraised  high  maturity  level  goals.  Overall,  the 
appraisers  rated  the  organizations’  documentation  very  high  for  this  activity,  with  two-thirds 
saying  it  was  good  or  excellent.  When  looking  at  whether  the  organizations  achieved  their 
maturity  level  goals,  the  disparity  in  documentation  shows  a  stark  contrast.  More  than  85  percent 
of  the  organizations  that  achieved  high  maturity  status  rated  the  documentation  as  good  or 
excellent,  while  only  30  percent  of  those  that  did  not  achieve  high  maturity  rated  the 
documentation  of  analytical  results  comparably  well. 


2009:  Description  of  the  Organization's  Documentation  for 
Process  Performance  and  Quality  Measurement  Results 
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Figure  6-18:  2009  —  Relationship  between  the  quality  of  process  performance  model 

documentation  and  achievement  of  appraised  high  maturity  level  targets 


With  few  exceptions,  the  relationships  between  the  various  aspects  of  organizational  context  that 
we  have  discussed  in  this  report  are  quite  similar  with  both  the  organizations’  achievement  of 
their  appraisal  high  maturity  level  goals  and  the  FIMLAs’  judgments  of  overall  value  added  by  the 
use  of  process  performance  models.  The  most  notable  difference  is  the  strength  of  the 
relationships  with  the  extent  to  which  the  HMLAs  reported  that  process  performance  model 
predictions  were  used  to  inform  decision  making  in  the  organizations’  status  and  milestone 
reviews.  Recall  from  Section  3  that  the  relationship  with  the  FIMLA’s  judgments  about  the  overall 


75  |  CMU/SEI-201 0-TR-022 


value  of  the  process  performance  models  is  extremely  strong  (gamma  =  .73).  As  shown  in  Figure 
6-19,  the  relationship  is  weaker  here,  although  still  quite  strong.  More  importantly  the  relationship 
is  similar  to  the  relationship  with  overall  value.  Organizations  that  achieved  their  appraisal  goals 
used  their  process  performance  models  much  more  frequently  in  their  status  and  milestone 
reviews  from  the  FIMLAs’  perspectives  than  the  organizations  that  failed  to  meet  their  maturity 
level  target  goals. 


2009:  How  often  are  process  performance 
model  predictions  used  to  inform  decision 
making  in  the  organization's  status  and 
milestone  reviews? 
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Figure  6-19:  2009  —  Relationship  between  use  of  process  performance  model  predictions  in 

status  and  milestone  reviews  with  achievement  of  appraised  high  maturity  level  targets 
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The  only  other  notable  difference  in  the  associative  relationships  between  the  two  y-variables 
(overall  value  of  process  performance  modeling  and  achievement  of  maturity  goal)  with  respect  to 
aspects  of  organizational  context  is  shown  in  Table  15.  The  relationships  with  stakeholder 
involvement  in  setting  goals  and  deciding  on  plans  of  action  for  measurement  and  analysis  are 
somewhat  weaker  for  customer  involvement  with  both  y-variables  and  for  involvement  of 
executive  and  senior  managers  with  achievement  of  the  organizations’  appraisal  target  goals.  As 
mentioned  earlier,  these  instances  may  be  a  statistical  artifact,  although  the  appraisers  may  have 
relatively  less  insight  about  these  two  stakeholder  groups  than  do  the  earlier  survey  respondents 
from  the  appraised  organizations  themselves. 

Table  15:  Comparison  of  relationships  between  stakeholder  involvement  with  achievement  of 
target  goals  and  overall  value  of  process  performance  model  outcomes 


Stakeholder  role 

Strength  of  Association  (y) 

Achievement 
of  appraisal 
target  goals 

Overall  value 
added  by 
process 
performance 
models 

Customers 

0.29 

0.38 

Executive  and  senior  managers 

0.30 

0.59 

Middle  managers  (e.g.,  program  or  product  line) 

0.62 

0.66 

Project  managers 

0.56 

0.59 

Project  engineers  and  other  technical  staff 

0.36 

0.33 

Process  and  quality  engineers 

0.39 

0.35 

Measurement  specialists 

0.46 

0.49 

Simple  summed  index  in  2009  HMLA  survey 

0.63 

0.75 
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The  strengths  of  association  for  almost  all  of  the  other  relationships  between  the  organizational 
context  x-variables  and  the  two  y-variables  are  essentially  the  same.  Only  two  aspects  of  the 
quality  of  measurement-related  training  differ  by  more  than  .10  in  Table  16  (coaches  and  mentors 
and  users  of  models).  They  may  vary  by  chance,  although  it  also  is  possible  that  the  appraisers  are 
less  likely  to  have  sufficient  information  to  answer  the  questions  with  confidence  [12]. 

Table  16:  Comparison  of  relationships  between  other  alignment  and  coordination  factors 
with  achievement  of  target  goals  and  overall  value  of  process  performance  model 

outcomes 


Strength  of  Association  (y) 

Other  factors 

Achievement 
of  appraisal 
target  goals 

Overall  value 
added  by 
process 
performance 
models 

Executive  and  senior  managers 

0.57 

0.66 

Middle  managers  (e.g.,  program  or  product  line) 

0.52 

0.56 

Project  managers 

0.61 

0.66 

Project  engineers  and  other  technical  staff 

0.66 

0.59 

Process  or  quality  engineers 

0.75 

0.76 

Process  performance  model  builders  &  maintainers 
(e.g.,  Six  Sigma  black  belts  or  other  measurement 
specialists) 

0.72 

0.69 

Coaches  &  mentors  who  assist  the  model  builders  and 
maintainers  (e.g.,  Six  Sigma  master  black  belts) 

0.78 

0.54 

Those  who  collect  and  manage  the  baseline  data  (e.g., 
Six  Sigma  green  belts,  other  project  engineers  or  EPG 
members) 

0.70 

0.77 

Users  of  the  models 

0.40 

0.56 

Modelers’  understanding  of  the  CMMI  definition  of  a 
process  performance  model 

0.75 

0.78 

Modelers’  understanding  of  the  CMMI  definition  of  a 
process  performance  baseline 

0.76 

0.72 

Modelers’  understanding  of  the  circumstances  when 
process  performance  baselines  are  useful 

0.77 

0.76 

Modelers’  understanding  of  the  circumstances  when 
process  performance  models  are  useful 

0.79 

0.71 

Managers’  understanding  of  the  process  performance 
model  results  that  they  use 

0.78 

0.83 

Availability  of  well-prepared  personnel  to  work  on 
process  performance  modeling 

0.74 

0.76 
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7  Summary  and  Conclusions 


As  was  true  in  our  previous  survey  of  high  maturity  organizations  in  2008,  all  of  the  certified 
HMLAs  in  the  2009  survey  sample  reported  that  the  organizations  they  appraised  or  coached 
found  business  value  in  the  use  of  process  performance  models  (see  Section  3).  They  most  often 
said  that  the  organizations’  models  were  very  valuable  and  that  they  had  “obtained  much  useful 
information  from  them”  (about  45  percent).  Almost  ten  percent  said  that  they  were  extremely 
valuable,  and  that  they  “couldn’t  do  their  work  properly  without  them.”  Others  said  that  their 
models  provided  mixed  value,  but  that  they  “have  obtained  useful  information  on  occasion”  (40 
percent),  and  very  few  reported  little  or  no  value  (5  percent).  As  was  true  in  the  2008  survey,  none 
of  the  HMLAs  reported  that  the  organizations  were  worse  off  as  a  result  of  their  process 
performance  modeling  efforts. 

More  importantly,  differences  in  the  appraisers’  perceptions  about  the  outcomes  of  the  modeling 
varied  consistently  as  a  function  of  their  judgments  about  how  well  the  “healthy  ingredients”  of 
process  perfonnance  models  were  understood  and  used.  The  relationships  also  differed  as  a 
function  of  the  extent  to  which  a  varied  mix  of  several  statistical  and  analytical  methods  was  used 
in  the  modeling  (see  Section  6.1.1).  This  was  true  for  both  surveys.  In  fact,  with  relatively  few 
exceptions,  the  comparable  statistical  relationships  were  stronger — sometimes  considerably 
stronger — in  the  survey  of  certified  HMLAs  than  they  were  in  the  earlier  survey  of  the  appraised 
organizations  themselves. 

Similarly,  as  is  typical  for  process  improvement  in  general,  the  overall  value  reported  varied  by 
how  much  management  support  existed  for  process  perfonnance  modeling  in  these  organizations 
(see  Section  6.1.2).  The  reported  value  of  the  modeling  according  to  the  appraisers  also  varied  as 
expected  by  several  other  measures  of  organizational  context,  and  alignment  and  coordination  of 
the  organizations’  measurement  and  analysis  activities.  Increased  value  was  associated  with 
higher  quality  training  programs,  better  understanding  by  managers  of  process  performance  model 
results,  availability  of  qualified  staff,  and  stakeholder  involvement  in  setting  goals  and  deciding 
on  plans  of  action  for  measurement  and  analysis  . 

Although  the  response  categories  sometimes  differed  between  the  two  surveys,  a  series  of 
questions  about  common  challenges,  barriers,  and  facilitators  of  successful  measurement  and 
analysis  also  varied  predictably  with  the  appraisers’  judgments  about  the  overall  value  of  the 
organizations’  process  performance  modeling  efforts  (Section  6.1.3). 

Throughout  this  report,  most  of  the  statistical  relationships  mentioned  are  quite  strong  for  survey 
data.  The  community  can  be  confident  that  the  appraisers’  judgments  are  consistent  with  the 
organizations’  own  views  of  the  value  of  measurement  and  analysis  to  their  work.  In  fact,  with 
relatively  few  exceptions,  the  statistical  relationships  were  stronger — sometimes  considerably 
stronger — in  the  survey  of  authorized  HMLAs  than  they  were  in  the  earlier  survey  of  the 
appraised  organizations  themselves.  In  addition,  many  of  the  same  factors  were  related  to 
whether  or  not  the  appraised  organizations  achieved  their  target  maturity  level  goals. 
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That  is  not  surprising  given  the  fact  that  the  organizations  that  achieved  their  appraisal  goals  also 
were  considerably  more  likely  to  gain  business  value  from  their  process  performance  modeling 
efforts  (Section  6.2).  Once  again,  the  statistical  relationships  are  extremely  strong  for  survey  data 
such  as  these. 


These  surveys  also  help  shed  light  on  a  difficult  problem  for  the  software  engineering  and 
development  community.  How  can  we  express  the  value  of  measurement  programs  in  concrete 
terms  without  violating  confidentiality  by  exposing  proprietary  information  that  organizations  are 
unwilling  to  share?  The  statistical  associations  presented  in  this  report  address  this  issue.  They 
demonstrate  a  solid  relationship  between  the  usefulness  of  measurement  and  analysis  and  the 
successful  achievement  of  high  maturity  appraisal  goals.  But  the  data  speak  to  more  than  enabling 
a  climb  up  the  maturity  ladder.  The  extent  of  alignment  for  so  many  of  the  analytical  activities 
within  successful  organizations  suggests  a  deeper  understanding  and  appreciation  for  the 
appropriate  use  of  data  to  inform  decision  making. 

The  empirical  results  presented  here  provide  evidence  for  organizations  that  wish  to  adopt 
effective  measurement  and  analysis  activities  in  general  and  useful  process  performance  modeling 
in  particular.  Most  of  the  factors  that  we  have  analyzed  in  both  surveys  and  that  are  associated 
with  the  successful  use  of  process  performance  models  are  controllable  by  management  and 
technical  actions.  These  results  should  contribute  to  more  efficient  and  effective  implementations 
and  use  of  measurement  and  analysis  techniques  and  consequently  better  performance  in  software 
engineering.  We  hope  that  others  will  be  encouraged  by  this  report  to  participate  actively  with  us 
in  gathering  further  evidence  for  the  wider  community  that  the  effective  use  of  measurement  and 
analysis  can  and  has  led  to  predictable  success  and  better  products. 
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Appendix  A  The  2009  Questionnaire  for  the  Survey  of 
HMLAs 


This  appendix  contains  a  listing  of  all  of  the  survey  questions  that  have  forced  choice,  “closed 
ended”  answers.  It  is  annotated  with  the  number  of  responses  for  each  answer  and  the  percentage 
of  the  total  answers  that  each  answer  represents,  except  for  the  multiple  response  questions  which 
present  only  the  counts. 
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Software  Engineering  Institute 

Carnegie  Mel  Ion 


The  State  of  Measurement  &  Analysis  2009: 

Survey  of  Applications  in  Support  of  High  Maturity  Practice 


I.  About  Yourself  &  the  Appraised  Organization 

1-1.  Which  of  the  following  best  describes  the  role  you  play  in  your  organization? 


Category 

Count 

% 

1  =  Lead  appraiser 

48 

57% 

2  =  Appraisal  team  member 

11 

13% 

3  =  High  maturity  practices  coach 

24 

29% 

No  response 

1 

1% 

Total 

84 

1-2.  Approximately  how  much  of  your  work  time  have  you  spent  during  the  past  year  in  each  of  the 
following?  ( Please  specify  an  approximate  percentage  for  each ) 

_ %  ...  High  Maturity  lead  appraiser 

_ %  ...  High  Maturity  appraisal  team  member 

_ %  ...  High  maturity  practices  coach 

1-3.  How  is  the  organization  best  described?  ( Please  select  one ) 


Category 

Count 

% 

1  =  Commercial  off  the  shelf  (e.g.,  shrink-wrap  or  custom  installation  of 
enterprise  solutions  such  as  SAP  or  Oracle) 

4 

5% 

2  =  Contracted  new  development,  (e.g.,  for  use  in  particular  product  lines 
or  other  novel  point  solutions=S  1Q3> 

29 

35% 

3  =  In-house  or  proprietary  development  or  maintenance 

14 

17% 

4  =  Defense  contractor 

23 

28% 

5  =  Other  government  contractor 

2 

2% 

6  =  Department  of  Defense  or  military  organization 

3 

4% 

7  =  Other  government  agency 

0 

0% 

8  =  Other 

7 

9% 

Total 

82 

1-4.  What  is  the  primary  focus  of  the  organization ’s  work?  (Please  select  one ) 


Category 

Count 

% 

1  =  Product  or  system  development 

60 

72% 

2  =  Maintenance  or  sustainment 

14 

17% 

3  =  Acquisition 

0 

0% 

4  =  Service  provision 

4 

5% 

5  =  Other 

5 

6% 

Total 

83 
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1-5.  What  kinds  of  engineering  are  major  parts  of  the  organization ’s  work?  (Please  select  as  many 
as  apply)  [83  respondents] 


Category_ Count 


1  =  Software  engineering 

83 

2  =  Systems  engineering 

44 

3  =  Hardware  engineering 

23 

4  =  Design  engineering 

23 

5  =  Test  engineering 

37 

6  =  Other 

3 

1-6.  In  what  country  is  the  organization  primarily  located? 

Category 

Count 

% 

1  =  United  States 

33 

40% 

2  =  Canada 

2 

2% 

3  =  China 

12 

15% 

4  =  France 

2 

2% 

5  =  Germany 

0 

0% 

6  =  India 

6 

7% 

7  =  Japan 

4 

5% 

8  =  Netherlands 

0 

0% 

9  =  United  Kingdom 

1 

1% 

1 0  =  All  Others 

20 

24% 

Total 

82 

1-7.  Approximately  how  many  full-time  employees  in  the  organization  work  predominantly  in 
software,  hardware  or  systems  engineering  (e.g.,  development,  maintenance,  acquisition  or 
provision  of  related  services)? 

Category 

Count 

% 

1  =  25  or  fewer 

1 

1% 

2  =  26-50 

4 

5% 

3  =  51-75 

3 

4% 

4  =  76-100 

7 

9% 

5  =  101-200 

8 

10% 

6  =  201-300 

15 

18% 

7  =  301-500 

8 

10% 

8  =  501-1000 

6 

7% 

9=  1001-2000 

8 

10% 

1 0  =  More  than  2000 

18 

22% 

Total  82 


1-8.  Did  the  organization  achieve  its  appraised  high  maturity  level  goal?  (Please  select  one) 


Category 

Count 

% 

1  =Yes 

49 

67% 

2  =  No 

24 

33% 

Total 

73 
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1-9.  To  the  best  of  your  knowledge,  what  is  the  current  maturity  level  of  the  organization?  ( Please 
select  one) 


Category 

Count 

% 

1  =  CMMI  Maturity  Level  3  or  lower 

18 

22% 

2  =  Close  To  Maturity  Level  4 

12 

15% 

3  =  CMMI  Maturity  Level  4 

4 

5% 

4  =  CMMI  Maturity  Level  5 

44 

54% 

5  =  Don’t  know 

4 

5% 

Total 

78 

II.  Measurement  Related  Training  &  Staffing 

II- 1.  How  would  you  best  characterize  the  measurement  related  training  that  is  available  in  the 
organization?  (Please  select  one  for  each ) 


Executive  and  senior  managers 


Category 

Count 

% 

1  =  Excellent 

8 

10% 

2  =  Good 

20 

26% 

3  =  Adequate 

24 

31% 

4  =  Fair 

14 

18% 

5  =  Poor 

11 

14% 

Total  78 


Middle  managers  (e.g.,  program  or  product  line) 


Category  Count  % 


1  =  Excellent 

16 

21% 

2  =  Good 

25 

32% 

3  =  Adequate 

17 

22% 

4  =  Fair 

13 

17% 

5  =  Poor 

7 

9% 

Total  78 


Project  managers 


Category  Count  % 


1  =  Excellent 

23 

29% 

2  =  Good 

29 

37% 

3  =  Adequate 

13 

16% 

4  =  Fair 

8 

10% 

5  =  Poor 

6 

8% 

Total 

79 
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Project  engineers  and  other  technical  staff 


Category 

Count 

% 

1  =  Excellent 

15 

19% 

2  =  Good 

26 

33% 

3  =  Adequate 

19 

24% 

4  =  Fair 

13 

16% 

5  =  Poor 

6 

8% 

Total 

79 

Process  or  quality  engineers 


Category  Count  % 


1  =  Excellent 

37 

47% 

2  =  Good 

25 

32% 

3  =  Adequate 

11 

14% 

4  =  Fair 

3 

4% 

5  =  Poor 

3 

4% 

Total  79 


Process  performance  model  builders  &  maintainers  (e.g.,  Six  Sigma  black  belts  or  other 
measurement  specialists) 


Category  Count  % 


1  =  Excellent 

39 

50% 

2  =  Good 

22 

28% 

3  =  Adequate 

11 

14% 

4  =  Fair 

2 

3% 

5  =  Poor 

4 

5% 

Total  78 


Coaches  &  mentors  who  assist  the  model  builders  and  maintainers  (e.g.,  Six  Sigma  master  black 
belts) 


Category  Count  % 


1  =  Excellent 

32 

44% 

2  =  Good 

26 

36% 

3  =  Adequate 

6 

8% 

4  =  Fair 

5 

7% 

5  =  Poor 

3 

4% 

Total  72 


Those  who  collect  and  manage  the  baseline  data  (e.g.,  Six  Sigma  green  belts,  other  project 
engineers  or  EPG  members) 


Category  Count  % 


1  =  Excellent 

29 

36% 

2  =  Good 

27 

34% 

3  =  Adequate 

17 

21% 

4  =  Fair 

3 

4% 

5  =  Poor 

4 

5% 

Total 

80 
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Users  of  the  models 


Category 

Count 

% 

1  =  Excellent 

8 

11% 

2  =  Good 

33 

44% 

3  =  Adequate 

24 

32% 

4  =  Fair 

5 

7% 

5  =  Poor 

5 

7% 

Total 

75 

II-2.  Approximately  how  many  people  in  the  organization  work  with  process  performance 
baselines  and  models  -  as  part  of  their  explicitly  assigned  work  efforts  ?  ( Please  specify  a  number 
for  each  ...or  type  DK  if  you  Don ’t  know ) 

_  Those  who  collect  and  manage  the  baseline  data  (e.g.,  Six  Sigma  green  belts,  other 

project  engineers  or  EPG  members) 

_  Those  who  build  and  maintain  the  models  (e.g.,  Six  Sigma  black  belts  or  other 

measurement  specialists) 

_  Those  who  mentor  or  coach  the  model  builders  and  maintainers  (e.g.,  Six  Sigma 

master  black  belts) 

_  Those  who  use  the  model  results  to  inform  their  decision  making 

II-3.  How  well  do  the  people  who  create  the  organization ’s  process  performance  models  and 
baselines  understand  the  intent  of  CMM I?  (Please  select  one  for  each) 

The  CMMI  definition  of  a  process  performance  model 


Category 

Count 

% 

1  =  Extremely  well 

16 

20% 

2  =  Very  well 

33 

41% 

3  =  Moderately  well 

12 

15% 

4  =  To  some  extent 

16 

20% 

5  =  Hardly  at  all 

4 

5% 

Total  81 


The  CMMI  definition  of  a  process  performance  baseline 


Category_ Count  % 


1  =  Extremely  well 

27 

33% 

2  =  Very  well 

27 

33% 

3  =  Moderately  well 

17 

21% 

4  =  To  some  extent 

7 

9% 

5  =  Hardly  at  all 

3 

4% 

Total 

81 
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The  circumstances  when  process  performance  baselines  are  useful 


Category 

Count 

% 

1  =  Extremely  well 

20 

25% 

2  =  Very  well 

26 

32% 

3  =  Moderately  well 

22 

27% 

4  =  To  some  extent 

10 

12% 

5  =  Hardly  at  all 

3 

4% 

Total  81 


The  circumstances  when  process  performance  models  are  useful 


Category_ Count  % 


1  =  Extremely  well 

13 

16% 

2  =  Very  well 

27 

33% 

3  =  Moderately  well 

22 

27% 

4  =  To  some  extent 

14 

17% 

5  =  Hardly  at  all 

5 

6% 

Total  81 


II-4.  How  well  do  the  managers  in  the  organization  who  use  process  performance  model  results 
understand  the  results  that  they  use?  ( Please  select  one ) 


Category_ Count  % 


1  Extremely  well 

5 

6% 

2  Very  well 

25 

32% 

3  Moderately  well 

32 

41% 

4  To  some  extent 

11 

14% 

5  Hardly  at  all 

6 

8% 

Total 

79 

II-5.  How  often  are  qualified,  well-prepared  people  available  to  work  on  process  performance 
modeling  in  the  organization  when  you  need  them  (i.e.,  people  with  sufficient  measurement 
related  knowledge,  competence,  and  statistical  sophistication)?  (Please  select  one) 


Category 

Count 

% 

1  =  Almost  always  (Greater  than  or  equal  to  80%) 

27 

33% 

2  =  Frequently  (Greater  than  or  equal  to  60%) 

18 

22% 

3  =  About  half  of  the  time  (Greater  than  40%  but  less  than  60%) 

15 

19% 

4  =  Occasionally  (Less  than  or  equal  to  40%) 

15 

19% 

5  =  Rarely  if  ever  (Less  than  or  equal  to  20%) 

6 

7% 

Total 

81 

87  |  CMU/SEI-201 0-TR-022 


II-6.  Does  the  organization  provide  promotion  or  financial  incentives  for  its  employees  that  are 
tied  to  the  deployment  and  adoption  of  measurement  and  analysis  (e.g.,  via  Six  Sigma  belt 
programs)?  ( Please  select  as  many  as  apply)  [66  respondents] 


Category 

Count 

No 

32 

Yes  ...  for  executive  and  senior  managers 

12 

...  for  middle  managers  (e.g.,  program  or  product  line) 

14 

...  for  project  managers 

12 

...  for  project  engineers  and  other  technical  staff 

9 

...  for  others  (Please  describe  briefly) 

7 

Don’t  know 

29 

III.  Alignment,  Coordination  &  Infrastructure 


III- 1 .  How  would  you  characterize  the  involvement  of  various  potential  stakeholders  in  setting 
goals  and  deciding  on  plans  of  action  for  measurement  and  analysis  in  the  organization?  ( Please 
select  one  for  each) 


Customers 


Category 

Count 

% 

1  =  Extensive 

4 

6% 

2  =  Substantial 

15 

21% 

3  =  Moderate 

19 

27% 

4  =  Limited 

20 

28% 

5  =  Little  if  any 

13 

18% 

Total 

71 

Executive  and  senior  managers 


Category 

Count 

% 

1  =  Extensive 

20 

27% 

2  =  Substantial 

30 

40% 

3  =  Moderate 

12 

16% 

4  =  Limited 

7 

9% 

5  =  Little  if  any 

6 

8% 

Total 

75 

Middle  managers  (e.g.,  program  or  product  line) 


Category 

Count 

% 

1  =  Extensive 

19 

25% 

2  =  Substantial 

23 

30% 

3  =  Moderate 

22 

29% 

4  =  Limited 

8 

11% 

5  =  Little  if  any 

4 

5% 

Total 

76 
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Project  managers 


Category 

Count 

% 

1  =  Extensive 

17 

22% 

2  =  Substantial 

30 

39% 

3  =  Moderate 

16 

21% 

4  =  Limited 

10 

13% 

5  =  Little  if  any 

3 

4% 

Total 

76 

Project  engineers  and  other  technical  staff 

Category 

Count 

% 

1  =  Extensive 

2 

3% 

2  =  Substantial 

16 

23% 

3  =  Moderate 

26 

37% 

4  =  Limited 

16 

23% 

5  =  Little  if  any 

12 

17% 

Total 

72 

Process  and  quality  engineers 

Category 

Count 

% 

1  =  Extensive 

25 

33% 

2  =  Substantial 

25 

33% 

3  =  Moderate 

20 

27% 

4  =  Limited 

4 

5% 

5  =  Little  if  any 

1 

1% 

Total 

75 

Measurement  specialists 

Category 

Count 

% 

1  =  Extensive 

27 

39% 

2  =  Substantial 

22 

31% 

3  =  Moderate 

11 

16% 

4  =  Limited 

6 

9% 

5  =  Little  if  any 

4 

6% 

Total 

70 

Others  ( Please  describe  briefly) 

Category 

Count 

% 

1  =  Extensive 

0 

0% 

2  =  Substantial 

1 

50% 

3  =  Moderate 

1 

50% 

4  =  Limited 

0 

0% 

5  =  Little  if  any 

0 

0% 

Total  2 
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III-2.  Which  of  the  following  best  describes  how  work  on  measurement  and  analysis  is  staffed  in 
the  organization?  (Please  select  as  many  as  apply) 


Category 

Count 

1  =  An  organization-wide,  division  or  similar  corporate  support  group 
(e.g.,  an  engineering  process,  quality  assurance  or  measurement 
group) 

50 

2  =  Separate  groups  or  individuals  in  different  projects  or  other 
organizational  units  (e.g.,  project,  product  team  or  similar  work  groups 

25 

3  =  A  few  key  people  (or  one  person)  in  the  organization  who  are 
measurement  experts 

24 

4  =  Other  (Please  describe  briefly) 

7 

III-3.  For  which  of  the  following,  if  any,  does  the  organization  provide  automated  support  for  its 
measurement  related  activities?  ( Please  select  as  many  as  apply  ...  or  be  sure  to  check  'None  of 
the  above'  if  appropriate) 


Category 

Count 

Data  collection  (e.g.,  online  forms  with  “tickler”  reminders,  time- 
stamped  activity  logs,  static  or  dynamic  analyses  of  call  graphs  or  run¬ 
time  behavior) 

55 

Commercial  work  flow  automation  that  supports  data  collection 

13 

Data  management  (e.g.,  relational  or  distributed  database  packages,  open 
database  connectivity,  tools  for  data  integrity,  verification,  or  validation) 

34 

Spreadsheet  add-ons  for  basic  statistical  analysis 

65 

Commercial  statistical  packages  that  support  more  advanced  analyses 

56 

Customized  spreadsheets  for  routine  analyses  (e.g.  for  defect  phase 
contaimnent) 

58 

Commercial  software  for  report  preparation  (e.g.,  graphing  packages  or 
other  presentation  quality  results) 

28 

Other 

2 
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III-4.  How  often  does  the  organization  do  the  following  with  the  data  it  collects?  (Please  check 
one  for  each ) 


Category _ Count 

Check  for  out  of  range  or  other  illegal  values  in  the  recorded  data  69 

Evaluate  the  number  and  distribution  of  missing  data  4 1 

Ensure  that  missing  data  are  not  inadvertently  treated  as  zero  values  49 

Check  for  precision  and  accuracy  of  the  data  53 

Estimate  measurement  error  statistically  34 

Check  for  inconsistent  interpretations  of  measurement  definitions  57 

Check  for  consistency/reliability  of  measurement  results  and 
procedures  across  time  and  reporting  units _ 56 

Check  for  consistency  of  classification  decisions  based  on  the  same 
information,  (otherwise  known  as  inter-coder  reliability)  29 

Analyze  &  address  the  reasons  for  unusual  patterns  in  the  data 
distributions,  e.g.,  outliers,  skewness,  or  other  aspects  of  non  normal 
distributions  60 

Analyze  &  address  the  reasons  for  unusual  or  unanticipated 
relationships  between  two  or  more  measures  46 

Automate  data  quality/integrity  checks  for  ease  of  collecting  consistent 
data _ 29 

Other  (Please  describe  briefly)  4 

None  of  the  above  2 

Don’t  know  5 


IV.  Use  of  Process  Performance  Models  &  Baselines 

IV- 1.  Which  of  the  following  product  quality  and  project  performance  outcomes  were  routinely 
predicted  with  process  performance  models  in  the  organization?  ( Please  select  as  many  as  apply 
...  or  be  sure  to  check  ‘None  of  the  above  ’  if  appropriate)  [77  respondents] 

Category _ Count 

Delivered  defects _ 65 

Type  or  severity  of  defects _ 35 

Product  quality  attributes  (e.g.,  mean  time  to  failure,  design  complexity, 

maintainability,  interoperability,  portability,  usability,  reliability, 

complexity,  reusability  or  durability) _ 23 

Quality  of  services  provided  (e.g.,  IT  ticket  resolution  time) _ 21 

Cost  and  schedule  duration _ 54 

Work  product  size  2 1 

Accuracy  of  estimates  (e.g.,  cost,  schedule,  product  size  or  effort) _ 39 

ROI  of  process  improvement  or  related  financial  performance  1 7 

Customer  satisfaction  23 

Other  (Please  describe  briefly)  9 

None  of  the  above  3 

Don’t  know  1 
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IV-2.  Which  of  the  following  (often  interim)  process  performance  outcomes  are  routinely 
predicted  with  process  performance  models  in  the  organization?  ( Please  select  as  many  as  apply 
...  or  be  sure  to  check  ‘None  of  the  above  ’  if  appropriate)  [76  respondents] 


Category 

Count 

Escaped  defects  (e.g.,  as  predicted  by  defect  phase  containment 
models) 

58 

Cost  of  quality  and  poor  quality  (e.g.,  rework) 

37 

Estimates  at  completion  (i.e.,  performed  periodically  throughout  the 
project) 

43 

Requirements  volatility  or  growth 

20 

Effectiveness  or  efficiency  of  inspection  or  test  coverage 

44 

Practitioner  adherence  to  defined  processes 

10 

Other  (Please  describe  briefly) 

1 

None  of  the  above 

4 

Don’t  know 

2 

IV- 3.  Which  of  the  following  processes  and  activities  are  routinely  modeled  in  the  organization? 
(Please  select  as  many  as  apply  ...  or  be  sure  to  check  ‘None  of  the  above  ’  if  appropriate)  [75 
respondents] 


Category 

Count 

Project  planning  and  estimation 

53 

Requirements  engineering 

33 

Product  architecture 

10 

Software  design  and  coding 

54 

Process  documentation 

7 

Quality  control  processes 

46 

Systems  engineering  processes 

13 

Hardware  engineering  processes 

5 

Acquisition  or  supplier  processes 

3 

Other  (Please  describe  briefly) 

8 

None  of  the  above 

2 

Don’t  know 

2 

IV-4.  How  much  emphasis  does  the  organization  place  upon  the  following  in  its  process 
performance  modeling?  ( Please  select  one  for  each) 

Accounting  for  uncertainty  and  variability  in  predictive  factors  and  predicted  outcomes 


Category 

Count 

% 

1  =  Extensive 

13 

18% 

2  =  Substantial 

28 

38% 

3  =  Moderate 

14 

19% 

4  =  Limited 

11 

15% 

5  =  Little  if  any 

7 

10% 

Total 

73 
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Factors  that  are  under  management  or  technical  control 


Category 

Count 

% 

1  =  Extensive 

14 

19% 

2  =  Substantial 

31 

42% 

3  =  Moderate 

14 

19% 

4  =  Limited 

12 

16% 

5  =  Little  if  any 

2 

3% 

Total  73 


Other  product,  contractual  or  organizational  characteristics,  resources  or  constraints 


Category_ Count  % 


1  =  Extensive 

4 

6% 

2  =  Substantial 

13 

20% 

3  =  Moderate 

24 

37% 

4  =  Limited 

16 

25% 

5  =  Little  if  any 

8 

12% 

Total  65 


Segmenting  or  otherwise  accounting  for  uncontrollable  factors 


Category_ Count  % 


1  =  Extensive 

9 

14% 

2  =  Substantial 

14 

22% 

3  =  Moderate 

19 

30% 

4  =  Limited 

11 

17% 

5  =  Little  if  any 

10 

16% 

Total  63 


Factors  that  are  tied  to  detailed  subprocesses 


Category_ Count  % 


1  =  Extensive 

10 

14% 

2  =  Substantial 

27 

39% 

3  =  Moderate 

20 

29% 

4  =  Limited 

4 

6% 

5  =  Little  if  any 

9 

13% 

Total 

70 

Factors  that  are  tied  to  larger,  more  broadly  defined  organizational  processes 


Category 

Count 

% 

1  =  Extensive 

9 

14% 

2  =  Substantial 

21 

32% 

3  =  Moderate 

20 

31% 

4  =  Limited 

10 

15% 

5  =  Little  if  any 

5 

8% 

Total 

65 
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Other  (Please  describe  briefly) 


Category 

Count 

% 

1  =  Extensive 

1 

25% 

2  =  Substantial 

2 

50% 

3  =  Moderate 

0 

0% 

4  =  Limited 

1 

25% 

5  =  Little  if  any 

0 

0% 

Total 

4 

IV- 5.  To  what  degree  are  the  organization ’s  process  performance  models  used  for  the  following 
purposes?  (Please  select  one  for  each ) 

Predict  final  project  outcomes 


Category 

Count 

% 

1  =  Extensive 

28 

37% 

2  =  Substantial 

25 

33% 

3  =  Moderate 

9 

12% 

4  =  Limited 

9 

12% 

5  =  Little  if  any 

5 

7% 

Total 

76 

Predict  interim  outcomes  during  project  execution  (e.g.,  connecting  “upstream”  with  “downstream” 
activities) 


Category_ Count  % 


1  =  Extensive 

18 

24% 

2  =  Substantial 

28 

37% 

3  =  Moderate 

19 

25% 

4  =  Limited 

4 

5% 

5  =  Little  if  any 

7 

9% 

Total 

76 

Model  the  variation  of  factors  and  understand  the  predicted  range  or  variation  of  the  predicted 
outcomes 


Category 

Count 

% 

1  =  Extensive 

16 

22% 

2  =  Substantial 

25 

34% 

3  =  Moderate 

16 

22% 

4  =  Limited 

12 

16% 

5  =  Little  if  any 

4 

5% 

Total 

73 
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Enable  “what-if”  analysis  for  project  planning,  dynamic  re-planning  and  problem  resolution 
during  project  execution 


Category 

Count 

% 

1  =  Extensive 

12 

16% 

2  =  Substantial 

20 

27% 

3  =  Moderate 

14 

19% 

4  =  Limited 

18 

25% 

5  =  Little  if  any 

9 

12% 

Total 

73 

Enable  projects  to  achieve  mid-course  corrections  to  ensure  project  success 


Category 

Count 

% 

1  =  Extensive 

9 

12% 

2  =  Substantial 

28 

38% 

3  =  Moderate 

22 

30% 

4  =  Limited 

11 

15% 

5  =  Little  if  any 

4 

5% 

Total 

74 

Other  (Please  describe  briefly) 


Category_ Count  % 


1  =  Extensive 

1 

20% 

2  =  Substantial 

0 

0% 

3  =  Moderate 

2 

40% 

4  =  Limited 

1 

20% 

5  =  Little  if  any 

1 

20% 

Total 

5 

V.  Other  Analytic  Methods  &  Techniques 

V-l.  To  what  extent  are  the  following  statistical  methods  used  in  the  organization ’s  process 
performance  modeling?  ( Please  select  one  for  each) 

Regression  analysis  predicting  continuous  outcomes  (e.g.,  bivariate  or  multivariate  linear 
regression  or  non-linear  regression) 


Category 

Count 

% 

1  =  Extensive 

25 

34% 

2  =  Substantial 

20 

27% 

3  =  Moderate 

17 

23% 

4  =  Limited 

6 

8% 

5  =  Little  if  any 

6 

8% 

Total 

74 
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Regression  analysis  predicting  categorical  outcomes  (e.g.,  logistic  regression  or  loglinear 
models) 


Category 

Count 

% 

1  =  Extensive 

4 

6% 

2  =  Substantial 

5 

8% 

3  =  Moderate 

23 

35% 

4  =  Limited 

18 

28% 

5  =  Little  if  any 

15 

23% 

Total 

65 

Analysis  of  variance  (e.g.,  ANOVA,  ANCOVA  or  MAN  OVA) 

Category 

Count 

% 

1  =  Extensive 

11 

15% 

2  =  Substantial 

19 

26% 

3  =  Moderate 

22 

30% 

4  =  Limited 

11 

15% 

5  =  Little  if  any 

11 

15% 

Total 

74 

Attribute  SPC  charts  (e.g.,  c,  u,  p,  or  np) 

Category 

Count 

% 

1  =  Extensive 

10 

14% 

2  =  Substantial 

25 

36% 

3  =  Moderate 

13 

19% 

4  =  Limited 

10 

14% 

5  =  Little  if  any 

11 

16% 

Total 

69 

Individual  point  SPC  charts  (e.g.,  ImR  orXmR) 

Category 

Count 

% 

1  =  Extensive 

33 

45% 

2  =  Substantial 

23 

31% 

3  =  Moderate 

10 

14% 

4  =  Limited 

4 

5% 

5  =  Little  if  any 

4 

5% 

Total 

74 

Continuous  SPC  charts  (e.g.,  XbarR  orXbarS) 

Category 

Count 

% 

1  =  Extensive 

17 

27% 

2  =  Substantial 

12 

19% 

3  =  Moderate 

8 

13% 

4  =  Limited 

15 

23% 

5  =  Little  if  any 

12 

19% 

Total 

64 
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Design  of  experiments 


Category _ Count  % 

1  =  Extensive _ 1  2% 

2  =  Substantial  3  5% 

3  =  Moderate  12  19% 

4  =  Limited _ 1 8  29% 

5  =  Little  if  any  28  45% 

Total  62 

Other  (Please  describe  briefly) 

Category _ Count  % 

1  =  Extensive _ 1  25% 

2  =  Substantial  0  0% 

3  =  Moderate _ 0  0% 

4  =  Limited  1  25% 

5  =  Little  if  any  2  50% 

Total  4 

V-2.  Which  of  the  following  visual  display  techniques  are  used  to  communicate  the  results  of  the 
organization ’s  analyses  of  process  performance  baselines?  (Please  select  as  many  as  apply  ...  or 
be  sure  to  check  ‘ None  of  the  above’  if  appropriate)  [78  respondents] 

Category _ Count 

Box  plots  45 

Histograms  62 

Scatter  plots  or  multivariate  charting  6 1 

Pareto  charts,  pie  charts  or  bar  charts  66 

Mosaic  charts  for  categorical  data  7 

Other  (Please  describe  briefly)  5 

None  of  the  above  2 

Don’t  know  3 

V-3.  Which  of  the  following  other  optimization  approaches  are  used  in  the  organization ’s  process 
performance  modeling?  (Please  select  as  many  as  apply  ...  or  be  sure  to  check  ‘None  of  the 
above  ’  if  appropriate)  [78respondents] 

Category _ Count 

Monte  Carlo  simulation  44 

Discrete  event  simulation  for  process  modeling  1 6 

Markov  or  Petri-net  models  1 

Probabilistic  modeling  30 

Neural  networks  0 

Optimization  9 

Other  (Please  describe  briefly)  3 

None  of the  above  13 

Don’t  know  9 
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V-4.  Which  of  these  decision  techniques  are  used  in  the  organization?  (Please  select  as  many  as 
apply. ..or  be  sure  to  check  ‘None  of  the  above  ’  if  appropriate)  [77respondents] 


Category 

Count 

Analytic  Hierarchy  Process  (AHP) 

4 

Real  options 

3 

Conjoint  analysis 

3 

Wide  band  Delphi 

40 

Weighted  multi  criteria  methods  (e.g.,  QFD  or  Pugh) 

47 

Decision  trees 

36 

Other  (Please  describe  briefly) 

3 

None  of  the  above 

5 

Don’t  know 

12 

VI.  Challenges  &  Value  Added 

VI- 1.  Following  are  a  series  of  statements  about  the  kinds  of  technical  challenges  that  projects 
sometimes  face.  How  well  do  they  describe  the  organization?  (( Please  select  one  for  each)  [76 
respondents] 


Statement 

Count 

Initial  project  requirements  are  not  well  defined 

29 

Requirements  change  significantly  throughout  the  life  of  the  projects 

42 

There  is  little  or  no  precedent  for  the  kind  of  work  we  are  doing 

12 

Significant  constraints  are  placed  on  product  quality  attributes  (e.g. 
reliability,  scalability,  security,  supportability,  etc.) 

37 

The  size  of  the  development  effort  is  large 

25 

The  technology  needed  for  the  projects  is  not  mature 

16 

There  are  extensive  needs  for  interoperability  with  other  systems 

36 

Insufficient  resources  (e.g.  people,  funding)  are  available  to  support  the 
projects 

23 

Insufficient  skills  and  subject  matter  expertise  are  available  to  support  the 
projects 

18 

Other  (Please  describe  briefly) 

8 

None  of  the  above 

4 

VI-2.  How  would  you  best  describe  the  organization's  documentation  of  its  process  performance  and 
quality  measurement  results  (e.g.,  in  the  organization 's  PIIDs,  measurement  repository  or  process 
asset  library)?  ( Please  select  one) 


Category 

Count 

% 

1  =  Excellent 

22 

29% 

2  =  Good 

29 

38% 

3  =  Adequate 

13 

17% 

4  =  Fair 

9 

12% 

5  =  Poor 

4 

5% 

Total 

77 
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VI-3.  Following  are  a  few  statements  about  the  possible  results  of  using  process  performance 
modeling.  To  what  extent  do  they  describe  the  organization  ?  (Please  select  one  for  each) 


Better  project  performance  (e.g.,  more  accurate  estimation,  reduced  cost,  shorter  cycle  time  or 
higher  productivity) 


Category 

Count 

% 

1  =  Almost  always 

22 

31% 

2  =  Frequently 

32 

44% 

3  =  About  half  the  time 

11 

15% 

4  =  Occasionally 

3 

4% 

5  =  Rarely  if  ever 

4 

6% 

Total  72 


Better  product  quality  (e.g.,  fewer  defects  or  improved  customer  satisfaction ) 


Category_ Count  % 


1  =  Almost  always 

29 

41% 

2  =  Frequently 

27 

38% 

3  =  About  half  the  time 

7 

10% 

4  =  Occasionally 

5 

7% 

5  =  Rarely  if  ever 

3 

4% 

Total  71 


Fewer  project  failures 


Category_ Count  % 


1  =  Almost  always 

21 

34% 

2  =  Frequently 

18 

29% 

3  =  About  half  the  time 

9 

15% 

4  =  Occasionally 

10 

16% 

5  =  Rarely  if  ever 

4 

6% 

Total  62 


Better  tactical  decisions  about  the  adoption  or  improvement  of  work  processes  and  technologies) 


Category_ Count  % 


1  =  Almost  always 

7 

11% 

2  =  Frequently 

29 

46% 

3  =  About  half  the  time 

3 

5% 

4  =  Occasionally 

16 

25% 

5  =  Rarely  if  ever 

8 

13% 

Total 

63 
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Better  strategic  decision  making  (e.g.,  about  business  growth  or  profitability) 


Category 

Count 

% 

1  =  Almost  always 

6 

10% 

2  =  Frequently 

20 

34% 

3  =  About  half  the  time 

6 

10% 

4  =  Occasionally 

17 

29% 

5  =  Rarely  if  ever 

10 

17% 

Total  59 


Other  ( Please  describe  briefly) 


Category_ Count  % 


1  =  Almost  always 

1 

50% 

2  =  Frequently 

1 

50% 

3  =  About  half  the  time 

0 

0% 

4  =  Occasionally 

0 

0% 

5  =  Rarely  if  ever 

0 

0% 

Total 

2 

VI-4.  How  often  are  process  performance  model  predictions  used  to  inform  decision  making  in 
the  organization 's  status  and  milestone  reviews?  ( Please  select  one) 


Category 

Count 

% 

1  Almost  always 

18 

25% 

2  Frequently 

25 

35% 

3  About  half  the  time 

6 

8% 

4  Occasionally 

10 

14% 

5  Rarely  if  ever 

12 

17% 

Total 

71 

VI-5.  Overall,  how  useful  have  process  performance  models  been  for  the  organization?  (Please 
select  one ) 


Category 

Count 

% 

1  =  Extremely  valuable — they  rely  on  them  extensively 

7 

9% 

2  =  Very  valuable — they  have  obtained  much  useful 
information  from  them 

34 

45% 

3  =  Mixed  value — they  have  obtained  useful  information  on 
occasion 

30 

39% 

4  =  Little  or  no  value 

4 

5% 

5  =  It’s  been  harmful,  not  helpful 

0 

0% 

6  =  Don’t  know 

1 

1% 

Total 

76 
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VII.  Barriers  &  Facilitators  of  Effective  Measurement  &  Analysis 


VII- 1 .  Which,  if  any,  of  the  following  have  been  major  obstacles  during  the  organization ’s 
journey  to  high  maturity?  (Please  select  as  many  as  apply  ...  or  be  sure  to  check  ‘None  of  the 
above  ’  if  appropriate)  [76  respondents] 


Statement 

Count 

They  focused  only  on  final  project  outcomes  rather  than  interim 
outcomes 

21 

They  didn’t  collect  data  frequently  enough  to  help  projects 
make  mid-course  corrections 

14 

They  failed  to  collect  enough  contextual  information  for  proper 
segmentation  and  stratification 

33 

They  failed  to  achieve  enough  consistency  in  their  measures  to 
aggregate  and  disaggregate  them  properly  across  the 
organization 

26 

They  failed  to  sufficiently  align  and  prioritize  their 
measurement  and  analysis  practices  with  our  business  and 
technical  goals  and  objectives 

15 

They  encountered  resistance  to  collecting  new  or  additional  data 
after  achieving  maturity  level  3 

18 

Their  management  thought  that  process  performance  modeling 
would  be  an  expensive  overhead  function  rather  than  an 
essential  part  of  project  work 

19 

They  spent  too  much  time  creating  reports  for  management 
review  instead  of  doing  thorough  analysis 

18 

They  emphasized  statistics  more  than  domain  knowledge  and 
ended  up  with  ineffective  models 

12 

They  didn’t  provide  sufficient  mentoring  and  coaching  for  the 
individuals  responsible  for  developing  the  models 

21 

Their  process  performance  modelers  don’t  have  sufficient 
access  to  people  with  statistical  expertise 

15 

Other  (Please  describe  briefly) 

12 

None  of  the  above 

16 

Don’t  know 

3 
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VII-2.  Following  are  a  series  of  statements  that  are  made  in  some  organizations  about  the  use  of 
process  performance  modeling.  Flow  well  do  they  describe  the  organization?  (Please  select  one 
for  each)  [76  respondents] 


Statement 

Count 

They  have  trouble  doing  process  performance  modeling  because  it  takes  too 
long  to  accumulate  enough  historical  data 

23 

Doing  process  performance  modeling  has  become  an  accepted  way  of 
doing  business  there 

35 

They  make  their  decisions  about  the  models  we  build  without  sufficient 
participation  by  management  or  other  important  stakeholders 

19 

They  have  trouble  convincing  management  about  value  of  doing  process 
performance  modeling 

19 

The  messenger  has  been  shot  for  delivering  bad  news  based  on  process 
performance  model  predictions 

6 

They  thought  they  knew  what  was  driving  process  performance,  but 
process  performance  modeling  has  taught  them  otherwise 

18 

Their  managers  want  to  know  when  things  are  off-track 

47 

Their  managers  are  less  willing  to  fund  new  work  when  the  outcome  is 
uncertain 

11 

They  use  data  mining  when  similar  but  not  identical  electronic  records  exist 

12 

They  do  real  time  sampling  of  current  processes  when  historical  data  are 
not  available 

24 

They  create  their  baselines  from  paper  records  for  previously  unmeasured 
attributes 

3 

Other  (Please  describe  briefly) 

2 

None  of  the  above 

6 

VII-3.  What  have  been  the  greatest  barriers  faced  by  the  organization  during  its  journey  to  high 
maturity?  What  have  you  done  to  overcome  them?  ( Please  describe  fully) — See  Appendix  B 


In  Conclusion 

What  kinds  of  special  difficulties,  if  any,  related  to  measurement  and  analysis  have  you 
experienced  in  appraising  or  coaching  organizations  that  aspire  to  high  maturity  status?  ( Please 
describe  fully) — See  Appendix  B 
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Appendix  B  Responses  to  the  Open  Ended  Questions 


This  appendix  contains  the  free-form  textual  answers  to  open-ended  questions  from  the 
questionnaire  for  the  survey  of  high  maturity  organizations.  Some  of  the  answers  have  been  edited 
for  purposes  of  nondisclosure. 
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II-6:  Does  the  organization  provide  promotion  or  financial  incentives  for  its  employees  that 
are  tied  to  the  deployment  and  adoption  of  measurement  and  analysis  (e.g.,  via  Six  Sigma 
belt  programs)?  -  Other 

excel.  Obviously,  employees  who  master  quantitative  concepts  are  more  likely  to  be  assigned  to 
interesting  projects  and  as  a  result,  have  more  chances  of  progressing  in  the  organization. 

The  “Don’t  know”  answer  applies  to  project  managers,  project  engineers,  and  others. 

Don’t  know  the  bonus  system,  based  on  award  fees 

The  customer  started  to  consider  the  CMMI  ML4  mid-late  last  year.  The  SEPG  members  did  not 
develop  enough  understanding  [of]  the  measurement  and  analysis  practices  to  move  toward  the 
ML4  level  improvement. 

We  equally  promote  and  incentivize  all  great  work  done  in  the  organization. 

The  awareness  and  need  is  increasing  with  time. 

For  the  Metrics  Council  members 

Some  but  they  are  few 

No  formal  specific  incentives  -  however  -  all  employees  are  rewarded  for  meeting  improvement 
goals(related  to  their  productive  use  of  M&A) 

Measurement  has  long  been  part  of  their  culture 

Heard  of  incentives  in  some  projects  which  are  linked  with  direct  project  success. 
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III-3:  For  which  of  the  following,  if  any,  does  the  organization  provide  automated  support 
for  its  measurement  related  activities?  -  Other _ 

proprietary  database  with  extensive  statistical  capabilities  (for  baselines,  weekly  project  data 
analysis,  statistical  testing,  etc.) 

extensive  library  of  both  automated  tools  and  support  for  measurement 
I  have  witnessed  a  mix  of  these 

In-house  developed  web-based  system  for  data  collection  and  analysis. 


III-4:  Which,  if  any,  of  the  following  does  the  organization  do  with  the  data  it  collects?  - 
Other _ 

May  do  more  than  checked 

The  boxes  checked  above  apply  to  aggregated  project  data  collected  by  the  organization  at 
infrequent  intervals.  For  event  level  data  collected  by  individual  projects,  the  answer  is  “Don’t 
know.” 

-  Compare  process  capability  baselines  within  and  across  project  types  after  appropriately 
grouping  them  to  ensure  homogeneity 

-  Establish  prediction  models  based  on  linear  regression  to  understand  causes  of  deviation  of 
actual  from  expected  when  such  associations/relationships  between  outcome  (dependent  variable) 
and  factors  (independent  variable)  are  possible 

-  Use  measurement  results  to  enable  proactive  goal  setting  for  the  development  process  at 
multiple  levels  and  lab-level  (or  domain-level)  stratification  within  the  organization 

-  Use  measurement  results  to  drive  causal  analysis  and  to  identify  process  improvement 
opportunities 

-  Use  statistical  techniques  to  demonstrate  capability  improvement 

They  do  normal  audits  to  check  measurement  data  quality — for  samples,  recreate  conditions  to 
verify  numbers  obtained 

We  did  not  look  into  all  this  detail  (although  we  did  some).  This  might  be  a  helpful  checklist  to 
use  on  appraisals. 
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IV-2:  Which  of  the  following  (often  interim)  process  performance  outcomes  were  routinely 
predicted  with  process  performance  models  in  the  organization?  -  Other _ 

May  have  done  others 

Productivity  (process  measure) 

First  pass  yield  (testing  effectiveness) 


IV-3:  Which  of  the  following  processes  and  activities  were  routinely  modeled  in  the 
organization?  -  Other 

I  assume  by  “modeled,”  you  are  referring  to  a  “process  performance  model”  -  like 
model/function. 

Verification  processes  /  peer  review  time  and  effort 
key  word  is  routine 
Verification  (peer  reviews) 

By  modeling,  what  I  mean  in  the  context  of  this  question: 

-  Use  of  abstractions  to  understand  the  intended  functionality  of  the  actual  behavior 

-  Enhancements  to  improve  the  understanding  obtained  using  working  prototypes  and  proof  of 
concepts 

-  Conducting  repeatable  “feasibility  studies”  to  refine  the  complexity  of  requirements  and  firm  up 
expected  and  intended  functionality 

-  Use  of  modeling  techniques  such  as  structured  methods,  Object  oriented  methods,  UML,  wire¬ 
frames  to  improve  the  clarity  of  analysis/design  of  the  software  solution 

-  Use  effective  traceability  tools  and  techniques  to  model  impact  of  requirements  changes 

Defects  found  in  doing  peer  reviews  on  work  products  like  specs  and  design  documentation  were 
also  used  in  predicting  meeting  quality  goals. 

estimation,  reviews  (peer  and  nonpeer),  coding 

Supplier  process  (delivery)  were  discussed,  but  no  model  exists  to  my  knowledge 

Test  engineering  processes 

Test 


IV-4:  How  much  emphasis  does  the  organization  place  upon  the  following  in  its  process 
performance  modeling?  -  Other 

Not  yet  always  able  to  differentiate  at  granularity  level  they  would  like  (e.g.,  different  types  of 
projects,  technologies,  domains,  etc.). 

Employees  are  characterizing  their  individual  performance  distribution  (average  and  std 
deviation),  which  is  an  input  into  models  using  this  data,  along  with  process  perfonnance 
baselines,  to  predict  project  outcomes. 

I  believe  that  the  word:  detailed  subprocesses:  needs  clarification/simplification! 

What  is  the  meaning  of  “detailed”? 

If  “detailed”  means:  focusing  at  the  root-cause  of  the  issue  causing  instability  -  then  that  is 
probably  justified  “ONLY  IN  AS  MUCH  AS,  FOR  IDENTIFICATION”  -  but  note  that  in 
software,  most  of  the  control  (meaning  measurement  opportunity)  is  pretty  much  at  a  macro  or 
process  level.  It  is  very  difficult  to  institutionalize  (nor  is  it  worth  the  while/money  spent)  to 
measure  at  such  low  levels  wherein  the  scope  for  individual  variation  is  very  extreme.  The 
usefulness  of  measures  at  such  micro  levels  (sub-sub-process  or  detailed  sub-process  level)  is 
fraught  with  dangers  of  inaccuracy,  approximation,  and  often  intrusive  to  yield  any  value. 
Besides,  one  must  consider  that  most  “detailed  subprocess”  measures  “cannot”  be  quantified. 
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IV-4:  How  much  emphasis  does  the  organization  place  upon  the  following  in  its  process 
performance  modeling?  -  Other 

Example:  how  does  one  quantify  design  complexity,  project  complexity,  clarity  levels,  individual 
ingenuity,  team  competency. . .? 


IV-5:  To  what  degree  are  your  organization's  process  performance  models  used  for  the 
following  purposes?  -  Other _ 

In  fairness  to  the  organization,  they  are  just  getting  started  in  their  level  4/5  process  definition 
work,  so  minimal  amount  of  “new”  practices  have  been  rolled  out  as  of  yet. 

Likelihood  of  obtaining  required  resources  from  management,  taking  into  account  organizational 
priorities  and  constraints,  is  one  of  the  factors  considered  during  modeling. 

The  reason  I  mark  “moderate”  and  not  “substantial/extensive”  is  because  attrition  plays  a 
significant  role  in  the  accuracy  of  prediction  achieved  in  software  processes.  Even  the  best  of 
high  maturity  process  work  cultures  have  attrition  rates  of  between  15-20%  per  year  losing  the 
cream  of  the  crop.  Project  metrics  are  highly  sensitive  to  “who”  is  working  on  the  project  -  even 
if  the  best  of  processes  is  a  given.  Having  a  mature  process  is  NO  substitute  for  “competent 
people.”  As  is  well  recognized  within  the  People  CMM  framework,  it  is  only  when  competent 
people  use  a  capable  process  can  you  get  predictable  outcomes. 

Most  measurements  suffer  from  too  many  unknowns  and  unpredictable  special  causes  -  example: 
cancelled  projects  due  to  re-prioritization  of  sponsorship  intent,  deferred  projects  or  functionality, 
uncertain  stakeholder  expectations,  stop-and-go  project  execution,  time-multiplexing  resources, 
and  individuals  on  multiple  projects. 

To  meet  the  requirements  of  CMMI  ML4  (not  that  this  is  a  valid  use,  but  this  org  would  tell  you, 
if  they  answered  honestly) 


V-l:  To  what  extent  are  the  following  statistical  methods  used  in  the  organization's  process 
performance  modeling?  -  Other 

Some  people  use  some  of  the  tools  for  product  analysis.  When  one  understands  the  key  words 
being  the  “process  performance  modeling”  the  answer  drastically  changes. 

Lognormal  model  is  used,  not  loglinear. 

DOE  has  very  limited  applicability  in  software.  “Quantifiable”  factors  and  limits  are  impossible 
to  define.  Example:  even  if  requirements  volatility  is  known  to  be  a  factor  causing  COPQ,  how 
does  one  quantify  the  associated  causes  say,  requirements  clarity,  complexity,  lack  of  domain 
training  ...  which  might  all  affect  the  req.  volatility  value? 

Monte  Carlo  Simulation 

Statistical  methods  are  not  really  yet  in  use  in  this  organization 


V-2:  Which  of  the  following  visual  display  techniques  are  used  to  communicate  the  results 
of  the  organization's  analyses  of  process  performance  baselines?  -  Other _ 

No  process  perfonnance  models  currently  in  real  use. 
control  charts  are  seen  most  of  the  time 
Control  charts 
process  control  charts 
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V-3:  Which  of  the  following  other  optimization  approaches  are  used  in  the  organization's 
process  performance  modeling?  -  Other _ 

Cannot  remember  exact  details  of  some  of  the  extended  interviews  -  been  almost  a  year 
No  process  perfonnance  models  currently  in  real  use. 

In  the  commercial  world  that  I  have  been  involved  in,  most  projects  are  short  duration  projects  (6 
to  8  months).  Many  projects  in  the  R&D  space  are  even  shorter  with  3-6  months  for  a  proof  of 
concept  or  feasibility  study  followed  by  implementation  in  the  next  2  to  3  months.  In  such  cases, 
where  the  domain  and  technology  is  being  tried  out  for  the  very  first  time,  and  an  opportunity  to 
repeat  the  same  trial  is  so  very  limited,  how  does  any  of  the  above  optimization  approaches  help? 

Optimization  techniques  make  sense  only  in  a  non-human  intensive  (or  machine  intensive),  mass 
production  of  a  single  product  (what  is  the  single  product  that  a  software  process  produces?),  with 
fixed  inputs  per  process  cycle  (most  unlike  in  any  software  process),  which  uses  a  fixed 
functionality  for  the  transformation  of  inputs  to  outputs  (most  unlike  any  software  process  given 
individual  and  team  competencies  coming  to  play)  and  the  output  characteristics  of  the  product 
are  defined  a  priori. 

Rayleigh  curves 

saw  Monte  Carlo  in  only  one  instance 


V-4:  Which  of  these  decision  techniques  are  used  in  the  organization?  -  Other _ 

maybe  others 

KT  Method  like  DAR  process. 

No  process  perfonnance  models  currently  in  real  use. 

F-test,  T-test  to  decide  on  formation  of  project  groups  for  PPBs  as  well  as  to  decide  if  process 
change  has  resulted  in  a  change  in  process  performance. 

Prediction  intervals. 

The  unchecked  techniques  may  be  used,  but  I  don't  have  personal  knowledge  of  it. 

SWOT 

SMART 


VI-1:  Following  are  a  series  of  statements  about  the  kinds  of  technical  challenges  that 
projects  sometimes  face.  Which  ones  typically  describe  this  organization?  -  Other _ 

projects  are  very  small,  mostly  minor  enhancements 

The  second  choice  is  the  one  that  happens  at  times  (requirements  changing  throughout  the 
project),  but  it  is  not  a  typical  occurrence.  Because  the  organization  operates  on  the  software 
factory  model,  projects  that  are  accepted  must  comply  to  what  the  factory  is  capable  of 
producing. 

Almost  every  factor  that  the  People  CMM  addresses  within  its  framework  that  is  related  to 
competencies  and  establishing  a  stable  work  environment. 

Lack  of  adequate  communication  and  understanding  of  the  impact  due  to  changes  to 
requirements. 

The  best  talent  and  competency  is  -  what  you  already  have!  So,  decisions  are  made  and  the 
consequences  lived  by  those  individuals,  who  to  an  external  eye  might  appear  grossly 
incompetent!  Example:  Have  you  ever  seen  a  knowledge  management  portal  being  developed 
using  Outlook?  I  am  aware  of  an  R&D  project  where  a  knowledge  management  portal  is  being 
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VI-1:  Following  are  a  series  of  statements  about  the  kinds  of  technical  challenges  that 
projects  sometimes  face.  Which  ones  typically  describe  this  organization?  -  Other _ 

conceived  as  an  “email  solution”  for  reasons  best  known  to  them. 

They  are  Chinese,  developing  for  Japanese  -  the  Japanese  are  the  ones  that  deal  with  the  real  end 
user. 

Initially,  some  of  the  projects  may  have  been  faced  with  requirements  that  were  not  well  defined. 

At  this  point,  most  of  the  projects  are  making  enhancements  to  existing  products,  and  in  these 

cases,  requirements  sometimes  change  a  bit  during  development. 

control  on  subcontractors  wrt  technical  issues 

Duration  and  cost  constraints 

Some  of  these  were  seen,  but  not  to  a  great  degree. 

Duration  of  the  project/product  is  typically  fixed. 


VI-3:  Following  are  a  few  statements  about  the  possible  results  of  using  process 
performance  modeling.  To  what  extent  do  they  describe  this  organization?  -Other _ 

Again,  org.  is  in  the  process  definition  phases  although  they  are  performing  some  rudimentary 
Level  4/5  practices  already.  Rollout  of  the  new/updated  practices  for  Level  4/5  won't  begin  until 
Q3  of  2009. _ 

My  view:  process  performance  models  and  PCBs  has  a  very  limited  shelf  life  for  any  “strategic 
decision  making”  to  result.  I  have  marked  “Frequently”  for  all  other  statements  with  the  proviso 
that  “its  applicability”  is  limited  in  context.  PPMs  and  PCBs  have  to  constantly  evolve  and 
change  as  the  organizational  dynamics  change. 

management  driven  decisions  rather  than  using  high  maturity  based  criterion 

Due  to  the  domain,  there  are  no  concerns  related  to  delivered  product  quality  (failures  in 
operation  are  not  allowed) 


VII- 1:  Which,  if  any,  of  the  following  have  been  or  still  are  major  obstacles  for  the 
organization?  -  Other 

Not  yet  able  to  have  granularity  desired  (i.e.,  have  not  been  able  to  identify  statistically  significant 
differences  in  projects,  technologies,  etc).  Have  not  yet  been  able  to  identify  as  many  variables  as 
they  would  like. 

difficulty  in  developing  PPMs  that  provide  real  business  value  given  the  type  of  work 
(applications  maintenance  and  small  enhancements)  under  increasing  resource  and  cost  pressures 

Failed  to  create  repeatable  procedures. 

Failed  to  translate  process  into  schedules.  The  schedules  are  product  based  with  no  attempt  to 
understand  the  process  or  process  step  associated  with  the  activity  in  the  schedule.  As  such 
measures  collected  are  schedule  or  product  based,  not  process  oriented. 

20%  of  the  staff  (including  all  project  managers)  had  specifically  been  trained  in  SPC,  probability 
and  statistics  at  a  local  university,  such  that  all  listed  obstacles  had  been  overcome  by  the  time  the 
appraisal  took  place.  One  obstacle  not  listed  is  that  people  at  a  lower  level  now  want  to  be  able  to 
better  understand  the  application  of  statistical  techniques,  even  though  they  may  not  have  to  apply 
them  routinely  in  their  work.  It  may  look  trivial,  but  it  has  the  potential  of  generating  a  perception 
that  employees  are  categorized  into  classes  based  on  the  knowledge  they  are  being  given. 

The  statistics  often  appear  to  state  already  known  facts. 

All  of  these  issues  came  in  as  they  started  toward  ML  4/5,  and  they  addressed  or  are  addressing 
all  of  them  since. 
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VII- 1:  Which,  if  any,  of  the  following  have  been  or  still  are  major  obstacles  for  the 
organization?  -  Other 

They  encountered  resistance  to  evolving  their  measurement  system  as  the  business  evolved. 

I  have  not  been  working  with  the  company  the  organization  was  a  part  of  since  the  beginning  of 
the  year...  and  that  organization  was  sold  to  another  company.  The  new  owner  has  high  maturity 
organizations  but  with  the  transition  their  status  relative  to  this  question  is  not  clear  to  me. 

Lack  of  appreciation/knowledge  of  senior  management  (specially  the  sponsor)  for  statistical 
knowledge.  I  recommend  that  the  sponsor  should  be  at  least  a  green  belt  6  sigma.  This  will 
educate  them  to  sponsor  and  support  and  drive  initiatives  for  the  teams  in  organization,  rather 
than  just  expressing  the  desire  of  maturity  status. 

Each  project  is  different  with  a  long  duration  (5  to  10  years).  It  is  difficult  to  get  consistent 
historical  data. 

They  put  too  much  attention  to  the  measures  directly  represent  their  final  and/or  interim 
objectives  rather  than  controllable  measures. 

They  may  be  measuring  too  much.  They  need  to  prioritize  their  performance  focus. 

I  Don’t  know  that  I  would  word  these  in  exactly  this  way,  but  if  it  was  close,  I  selected  it.  For 
example,  I  think  it  is  a  problem  that  models  are  not  seen  by  management  as  an  essential  part  of 
project  work. 

-  They  used  external  people  to  help  them  develop  the  models 

-  There  is  a  lack  of  appropriated  tools  to  help  on  statistical  aspects  and  also  to  make  the  MA 
process  more  automated 

Some  of  the  more  innovative  models  are  used  in  only  a  few  projects. 

The  organization  felt  that  there  is  a  dilemma  between  using  high  maturity  concepts  and 
methodologies  to  gain  useful  data  and  information  to  make  decisions  versus  meeting  the  SEI's 
CMMI  model  interpretation  expectations. 


VII-3:  What  have  been  the  greatest  barriers  faced  by  this  organization  during  its  journey  to 
high  maturity?  What  have  they  done  to  overcome  them?  (Please  describe  fully) 

They  were  high  maturity  8  years  ago  and  continue  to  improve  over  time 

Trying  to  focus  too  much  on  what  is  “easy”  (e.g.,  using  peer  review  data)  than  on  what  is  really 
“meaningful”  -  figuring  out  what  other  critical  indicators  might  be  useful  as  leading  indicators  of 
future  outcomes  of  interest. 

Perceived  change  in  high  maturity  requirements  at  SEI  since  their  first  ML5  appraisal  in  2005. 
They  reacted  by  sending  their  internal  personnel  to  SEI  high  maturity  classes  prior  to  their 
reappraisal  in  2008. 

While  good  data  is  collected  and  is  properly  analyzed,  it  typically  is  not  felt  to  be  significant  or 
appropriate  enough  to  change  decision-making  behavior.  Sometimes,  after  the  fact,  they  will 
realize  that  the  data  pointed  them  in  a  direction  they  didn't  take  because  PM  “instinct”  told  them 
otherwise. 

Cost  of  HM  activities  and  uncertain  payback.  While  there  has  been  some  value  from  HM,  it  has 
not  proven  to  cover  the  cost  of  training,  development,  data  collection,  analysis,  modeling,  etc.  and 
therefore,  cannot  be  sold  to  management  or  other  parts  of  company.  The  primary  driver  for  HM 
activities  was  contractual  commitment  to  the  client  (for  ACHIEVING  CMMI  L5,  rather  than 
delivering  clear  and  proven  business  value  from  CMMI  L5) 

As  the  total  strength  of  this  organization  was  more  than  3,500,  it  was  little  bit  challenging  to 
cover  the  total  population  for  face-to-face  CMMI  HML  related  training  and  hence  it  was 
encouraged  to  use  QPM  -  CBT. 

How  complex  are  the  PPMs  supposed  to  be  built?  Do  they  need  to  consider  all  possible  factors  or 
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VII-3:  What  have  been  the  greatest  barriers  faced  by  this  organization  during  its  journey  to 
high  maturity?  What  have  they  done  to  overcome  them?  (Please  describe  fully) _ 

just  enough  satisfy  their  business  needs?  say  the  “ideal  world”  and  “real  world.” 

Usually,  only  a  single  person  in  the  organization  is  the  measurement  expert;  not  having  a 
workflow  automation  tool  that  supports  data  collection.  The  organization  has  initiated  an  intense 
training  program  covering  various  roles;  including  contracting  a  coach  as  a  mentor  and  to  verify 
and  validate  the  approach  and  its  implementation.  The  organization  is  also  piloting  an  integrated 
project  management  systems  that  will  provide  workflow  automation  and  supports  data  collection. 

statistical  understanding  of  project  managers  and  management 

Dependence  on  an  external  consulting  company  to  help  the  organization  “implement”  the  CMMI 
high  maturity  practices  that  focused  more  on  developing  the  evidence  that  would  be  needed  as 
proof  to  achieve  ML  4  than  actually  helping  the  organization  to  focus  on  achieving  realistic 
business  goals  using  high  maturity  concepts.  I  do  not  know  what  they  have  done  to  overcome  this 
as  I  was  “fired”  as  their  lead  appraiser  after  the  conduct  of  their  SCAMPI  readiness  review  and 
the  appraisal  was  aborted. 

ft  has  been  and  will  remain  the  greatest  challenge  to  gain  sufficiently  deep  understandings  of  the 
meanings  of  CMMI  high  maturity  practices,  among  senior  managers  and  project  managers  within 
the  organization. 

Lack  of  documented  examples. 

Initially  just  setting  realistic  expectation  of  their  Level  4/5  process  journey  in  allowing  enough 
time  for  institutionalization  once  development  is  completed  -  time  will  tell. 

A  minimalistic  implementation  of  the  practices  from  the  process  areas  that  at  ML2  and  ML3,  and 
having  this  minimalistic  approach  validated  by  lead  appraisers  and  HMLAs. 

Areas  that  require  a  more  robust  implementation  have  been  pointed  out. 

Way  too  many  measures  initially.  They  had  to  consolidate  and  assess  the  importance  of  the 
measures  they  had  defined  and  measurement  data  they  were  collecting.  There  was  a  tendency  to 
measure  just  in  case  the  measurement  data  could  be  useful  in  the  future. 

Lack  of  knowledge  of  statistical  techniques  and  how  to  use  them  in  software  projects. 

They  have  undergone  training  and  have  got  many  of  the  project  managers  to  use  statistical  tools 
and  do  their  own  analysis  of  data  (rather  than  have  a  central  group  do  it  for  them). 

1.  Deriving  and  demonstrating  the  business  value  of  high  maturity  practices.  Exercise  of  using 
Goal  Decomposition  matrix  facilitated  a  lot  to  drill-down  the  high  level  business  goals  to  the 
lower  level  sub-processes  and  to  understand  their  relationships. 

2.  Understanding  the  benefits  for  implementing  statistical  tools  and  techniques  as  this  requires  an 
investment  on  tools  and  experts.  Continuous  dissemination  and  awareness  programs  focused  with 
practical  case  studies  helped  the  organization’s  leadership  team  to  understand  the  key  benefits 
statistical  tools  can  bring  to  connect  with  the  business  goals  on  an  on-going  basis. 

3.  Project  leaders  and  managers’  ability  to  get  in  to  the  habit  of  data-driven  approach  in  high 
maturity  environment.  This  posed  a  greater  challenge  as  this  required  focused  group  sessions, 
mentoring  support  to  bring  the  habit  of  using  data  for  decision  making.  Transfonnation  from 
perception  /  intuition  based  decision  making  to  fact  and  data  driven  decision  making  took  longer 
time  than  expected. 

4.  Above  all  the  transformation  from  ML3  scenario  to  ML4  scenario  is  more  of  a  cultural 
progression  and  the  team  including  the  leadership  management  did  not  realize  the  efforts  required 
to  do  this.  Also  efforts  need  to  be  expended  from  the  perspective  of  not  only  installing  the  high 
maturity  practices  but  also  sustaining  them  continuously  in  changing  turbulent  times. 

Recognizing  this  as  a  process  culture  and  impacts  every  function  in  the  Organization  would  be 
the  key  while  embarking  on  this  initiative. 
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VII-3:  What  have  been  the  greatest  barriers  faced  by  this  organization  during  its  journey  to 
high  maturity?  What  have  they  done  to  overcome  them?  (Please  describe  fully) _ 

This  organization  has  always  been  very  disciplined  from  a  process  perspective  (the  nature  of  their 
business  dictates  this).  They  had  years  of  historical  data.  When  they  started  analyzing  data  at  the 
true  subprocess  level,  they  discovered  problems  with  data  integrity  and  operational  definitions. 

The  greatest  barrier  in  my  opinion  has  been  a  mindset  that  assigns  QPM  and  CAR  to  projects  and 
OPP  and  OID  to  the  organization  at  a  much  higher  level  (i.e.,  the  organization  I  work  with  is  one 
division  within  a  business  sector.  OPP  and  OID  are  performed  at  the  sector  level).  The  result  is  a 
lack  of  understanding  at  the  project  level  of  the  interactions  among  these  process  areas,  and  a  lack 
of  focus  and  resources  at  the  sector  level  to  support  individual  project  needs  adequately.  There 
has  been  discussion  within  the  process  management  group  at  the  division  level  about  assuming 
more  of  the  OPP  and  OID  functions  and  better  integrating  all  of  the  high  maturity  activities  across 
the  projects  and  division,  but  no  effective  actions  have  resulted  yet. 

A  second  barrier  is  a  tendency  of  project  management  to  treat  CMMI  as  a  function  of  the  quality 
and  process  staff  at  the  project  level,  rather  than  as  an  integral  part  of  the  job  of  each  member  of 
the  project  team.  This  is  combined  with  a  focus  by  the  quality  and  process  staff  on  process 
compliance  and  on  passing  appraisals.  The  result  of  these  factors  with  regard  to  high  maturity 
practices  is  that  the  quality  and  process  staff  are  often  the  ones  performing  statistical  analysis  on 
data,  and  this  analysis  is  being  used  more  to  produce  artifacts  for  appraisals  than  to  drive  project 
management  decisions.  This  is  recognized  as  a  problem  by  the  organization  level  process 
management  group,  which  has  taken  actions  such  as  training  of  project  managers  and  project 
level  quality  and  process  staff,  as  well  as  regular  internal  appraisals  of  CMMI  practice 
implementation  with  feedback  to  project  management  on  the  weaknesses  resulting  from  these 
factors. 

Although  this  is  quite  a  large  organization  it  still  retains  (and  highly  values)  a  “startup”  culture 
that  is  contemptuous  of  formal  methods  of  any  kind.  Consequently  when  process-based 
approaches  are  expected,  the  result  is  often  pro  forma  execution  or  malicious  compliance.  This  is 
changing  in  departments  with  strong,  progressive  leaders,  but  these  individuals  are  still  in  the 
distinct  minority,  The  situation  is  further  complicated  by  a  reward  system  that  preferentially 
rewards  heroes  who  do  successful  firefighting.  Efforts  are  underway  across  the  organization  to 
change  this  system  to  one  that  rewards  process  based  behaviors  instead.  Much  work  remains  to  be 
done. 

Skepticism  of  the  value  of  ML  4/5  and  constant  churn  (perceived)  reinterpretation  by  the  SEI  in 
terms  of  what's  required  for  ML  4/5. 

understanding  PPMs  and  PPBs  and  using  statistical  management  in  development  projects.  This  is 
not  normally  done  or  written  about. 

It  is  a  completely  new  way  of  thinking  for  software  project  managers  and  senior  managers. 

Time  was  a  major  factor,  but  with  huge  workforce,  this  was  a  non-issue  later.  The  organization 
had  deployed  a  good  amount  of  resources  in  this  direction  along  with  tools. 

-  Getting  every  nook-and-cranny  of  the  organization  sensitive  to  the  importance  of  metrics-based 
decision  making. 

-  Driving  a  sense  of  “guilt”  and  an  of  emotion  of  being  ashamed  to  produce  a  lousy  job 

-  Behavioral  changes  required  to  sustain  high  process  maturity 

There  are  many  other  soft  factors  which  come  to  play  and  it  is  difficult  to  overcome  them  since 
the  sense  and  constancy  of  purpose  required  to  inculcate  “high  process  maturity”  waxes  and 
wanes  at  all  levels. 

the  greatest  barriers  faced:  types  of  project  are  variable;  staff  turnover;  skill  level 

they  have  done  to  overcome  them:  training,  classify  types  of  their  projects  and  lifecycle,  and  so 
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VII-3:  What  have  been  the  greatest  barriers  faced  by  this  organization  during  its  journey  to 
high  maturity?  What  have  they  done  to  overcome  them?  (Please  describe  fully) 

on 

The  greatest  barrier  has  been  uncertainly  in  the  sponsorship  area  due  to  reorganization  and 
integration  issues  within  the  larger  corporate  business  unit.  This  was  overcome  by  forging  a 
united  front  amongst  the  Engineering  and  QA  leadership  to  make  a  strong  case  for  fully  funding 
and  sponsoring  process  improvement  efforts. 

SEPG  members  have  not  sufficiently  developed  their  leadership  for  engineers  and  management 
on  the  M&A  to  improve  process  and  quality  performance. 

Effective  training  on  basic  SPC  with  relationship  to  process  may  be  needed  [by]  every  member. 

Normal  barriers-  the  occasional  need  to  replace  key  people  and  integrate  new  people  into  the 
organization;  communicating  with  changing  stakeholders  before,  during  and  after 
reorganizations;  setting  a  social  context  that  endorses  process  modeling  and  the  use  of 
measurements  in  daily  work 

The  organization  is  very  active  in  the  CMMI  community  and  has  participated  in  the  ongoing 
research  and  development  of  CMMI  and  high  maturity,  it  is  a  leading  edge  company 

The  second  largest  challenge  was  to  consider  sub  process  variation  as  part  of  the  models,  through 
the  use  of  deterministic  approaches,  avoiding  the  interpretation  of  statistical  distribution  of  sub 
processes  contributing  to  the  PPM's. 

Basic  data/measurement  definition,  collection  (deciding  what  to  measure  and  how  to  collect)  in 
the  midst  of  ongoing  complex  systems  development  with  demanding  customers 

The  greatest  barrier  historically  is  collect  actual  data  for  analysis.  They  have  worked  very  hard  to 
overcome  this  barrier. 

Personally,  I  think  they  want  so  much  to  be  “perfect”  that  they  have  limited  what  they  have  tried. 

This  is  a  very  large  organization,  but  only  a  small  part  of  it  is  involved  in  systems  development. 
Most  of  the  work  is  staff  augmentation  for  clients,  white  papers,  studies,  and  the  like.  They  have 
only  a  few  development  projects  that  use  the  organization's  process,  with  the  rest  of  the  systems 
development  effort  being  performed  using  processes  and  tools  specified  by  the  customer  for  that 
project;  hence,  there  is  not  a  lot  of  motivation  by  management  to  buy  automated  data  analysis 
tools  like  DataDrill,  for  example.  Most  of  the  M&A  and  HM  work  is  supported  using  Excel 
templates  developed  for  that  purpose.  A  few  people  who  have  been  through  6-sigma  training  have 
SigmaXL  and  Minitab,  and  the  bulk  of  the  responsibility  for  process  performance  analysis  and 
prediction  is  fielded  by  these  individuals. 

Trying  to  interpret  what  SEI  is  asking  for  in  high  maturity.  The  organization  set  a  goal  to  have  all 
exempt  engineers  trained  and  certified  as  Six  Sigma  Green  belts  and  then  to  employ  their 
expertise  within  their  area  of  project  development.  The  organization  has  experienced  both  great 
successes  and  great  failures  with  this  approach.  For  example,  it  is  fairly  easy  to  apply  QPM  to  the 
peer  review  process  and  increase  defect  containment.  However,  tackling  the  difficult  issues  in 
translating  needs  into  requirements  or  requirements  into  a  functional  design  do  not  lend 
themselves  to  statistical  process  control.  There  is  very  little  solid  statistical  data  that  can  be 
extracted  and  used  to  reliably  predict  future  performance. 

1.  Understanding  the  importance  of  sub-processes  and  controllable  factors:  Expectation 
clarification  sessions  with  lead  appraiser 

2.  Difficulty  of  modeling  work:  Used  a  six-sigma  black  belt  outside  of  the  organization  unit 
(within  their  own  org) 

3.  Getting  correct  predictions  from  the  models  :  Time  to  time  evaluations  and  fine -tuning  as 
necessary 

As  a  team  member,  not  involved  in  its  improvement  cycle,  so  I  do  not  know  the  detail  barriers. 

The  use  of  statistical  techniques  for  process  performance  modeling  and,  subsequently,  for 
statistically  managing  the  sub  process  performance  was  initially  the  biggest  barrier.  They 
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VII-3:  What  have  been  the  greatest  barriers  faced  by  this  organization  during  its  journey  to 
high  maturity?  What  have  they  done  to  overcome  them?  (Please  describe  fully) _ 

overcame  this  by  extensive  knowledge  acquisition,  finding  practical  and  sensible  ways  to  use 
these  techniques  and  piloting  in  a  few  projects  first  and  showing  that  it  really  works. 

The  other  challenge  was  in  identifying  the  critical  sub  processes.  Initially,  they  were  floundering 
but  when  the  process  models  started  to  give  them  useful  insights  into  the  behavior  of  the 
processes,  identifying  sub  processes  became  relatively  easier.  Other  barriers  pertained  to  the 
availability  (or  otherwise)  of  good,  reliable  data.  This  continues  to  be  a  problem  which  has  to  be 
continuously  tackled.  Given  the  wide  variety  of  projects  (and  the  large  number  of  projects),  this 
problem  crops  up  periodically.  The  Metrics  Council  has  to  tackle  this  in  various  ways  all  the  time. 

The  organization  I  am  speaking  for  has  just  started  the  journey  and  at  this  stage,  their  bigger 
barrier  is  the  difficulties  to  understand  what  a  PPM  is  and  the  sub-process  concept  and  value 
added 

1 .  Lack  of  appreciation/knowledge  of  senior  management  (specially  the  sponsor)  for  statistical 
knowledge.  I  recommend  that  the  sponsor  should  be  at  least  a  Green  belt  6  sigma  -  This  will 
educate  them  to  sponsor  and  support  and  drive  initiatives  for  the  teams  in  organization,  rather 
than  just  expressing  the  desire  of  maturity  status. 

2.  False  estimations  of  duration  for  high  maturity  outcomes.  If  the  concern  1  above  is  managed, 
then  sponsors  shall  have  a  better  understanding  of  the  high  maturity  intent  and  process,  thus  do  a 
proper  estimates  for  process  improvement  initiatives.  Any  underestimates  shall  put  false  criterion 
(like  date  of  ML  achievement)  for  the  success  of  the  initiative. 

3.  Inadequate  budget  for  the  journey  to  high  maturity  is  witnessed  wrt  allocation  of  money, 
people,  resources,  and  tools. 

4.  SEI  communications:  SEI  is  concerned  about  direct  ML  4-5  appraisals  (without  prior  ML  2-3 
records).  SEI  should  clearly  set  a  mandate  on  SEI  site  as  ML3  as  a  prerequisite  for  any  high 
maturity  appraisal.  Leading  an  initiative  includes  ability  and  intent  for  clear  communications  to 
all.  Else  the  sponsor  still  nourishes  the  desires  to  invest  in  direct  high  maturity  appraisals  and  land 
up  in  false  starts  with  their  internal  team,  lead  appraisers  and  SEI. 

The  domain  (duration  of  projects,  no  repetitive  projects) 

They  shall  reduce  the  scope  of  using  high  maturity  practices  to  verification  activities  (test, 
reviews...) 

Controllable  measures  have  not  considered  and  collected  well.  They  are  still  trying  to  overcome 
it. 

Lack  of  understanding  of  PPM,  statistical  techniques.  Lack  of  tools  to  compile  and  analyze  the 
measures.  Lack  of  management  commitment.  Management  just  says  “do”  but  there  is  no  follow¬ 
up  to  ensure  improvements  are  performed. 

Not  understanding  the  difference  between  CMMI  v  1.1  and  v  1.2  for  the  increased  understanding 
of  ML  4/5  PAs.  This  org  was  certified  at  ML  5  with  vl.l  and  when  it  became  time  to  update  their 
3  year  cert  it  was  vl.2  that  was  out.  They  had  a  real  hard  time  putting  PPOs,  PPMs,  and  PPBs  in 
practice.  They  are  to  undergo  their  appraisal  later  this  year. 

It  is  the  diversity  of  the  projects.  The  challenge  is  to  find  data  that  applies  to  most  projects  that 
then  can  be  used  to  drive  PPBs  and  PPMs. 

Maintaining  the  quality  of  staff  to  optimize  the  use  of  process  performance  modeling  activities, 
i.e.,  Six  Sigma  Black  Belts  and  Master  Black  Belts. 

The  fact  that  models  are  not  clearly  a  value  added  activity  for  processes  that  are  not  repetitive  and 
are  costly  for  answers  that  are  usually  obvious  with  other  kinds  of  analysis. 

Variability  in  predictions  is  still  larger  than  desired,  although  the  reduction  in  variability  is  a  focal 
point  at  present.  This  is  mitigated  by  proper  balance  between  quantitative  and  qualitative  analysis. 
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VII-3:  What  have  been  the  greatest  barriers  faced  by  this  organization  during  its  journey  to 
high  maturity?  What  have  they  done  to  overcome  them?  (Please  describe  fully) _ 

Getting  project  and  middle  management  engaged  in  quantitative  management. 

To  overcome  this  required  significant  support  (hand  holding)  during  deployment.  A  key 
individual  doing  the  analysis  had  very  creative  (humorous)  ways  of  presenting  analysis  results 
and  moving  senior  management  to  focus  on  solutions  required  to  get  projects  back  on  track, 
rather  than  finger  pointing.  Project  Management  now  sees  the  value  in  the  early  indicators  of 
probable  outcomes  so  than  can  increase  the  probability  of  positive  outcomes.  They  also  find 
themselves  in  better  negotiating  positions  with  customers  regarding  costs  and  schedule. 

This  organization  was  previously  appraised  by  another  HMLA  and  a  team  to  CMMI  ML  4  but 
they  have  significant  gaps  in  what  is  really  expected  at  ML  4,  therefore  convincing  the 
organization  that  their  gaps  are  very  significant  has  been  a  challenge.  Additionally,  moving  the 
organization  toward  operating  at  ML4  with  appropriate  universal  involvement  of  project 
members,  as  opposed  to  only  an  overhead  group  involved  in  QM  activities  has  been  very 
challenging  in  this  organization. 

Learning  the  statistical  methods  and  when  they  apply. 

Getting  management  to  believe  that  software  processes  can  be  statistically  managed  and  that 
effort  is  worth  the  cost. 

Prioritizing  their  performance  processes  and  measures.  The  senior  management  want  to  measure 
everything  and  the  project  leaders  complained  about  it  being  too  much  of  a  burden.  I  advised 
them  that  not  every  PA  needs  to  have  a  set  of  subprocesses  and  measures  (which  senior 
management  has  stated  as  a  future  goal). 

The  journey  took  longer  than  they  anticipated.  The  organization  persevered  in  their  data 
collection  and  analysis  efforts  and  continued  providing  technical  tools,  training,  and  support  to 
assist  the  projects. 

They  focus  the  ML  achievement  not  the  benefits  related  to  the  ML5.  They  have  to  change  their 
internal  culture  and  focus  to  understanding  their  business  and  performance  in  order  to  have 
improvements  and  better  results. 

lack  of  knowledgeable  and  skilled  resources,  who  can  apply  CMMI  concepts  to  achieve  tme 
business  value. 

Most  organizations  (and  the  ones  I  referred  here)  are  still  not  really  convinced  about  the  value  of 
all  this.  They  also  think  it’s  very  academic  and  theoretical  good  enough  to  pursue  a  PhD  and  only 
for  Professors  who  are  retired  or  about  to  retire.  The  extreme  “theory”  especially  with  the  release 
of  CMMI  vl.2,  and  not  knowing  what  really  is  expected  by  SEI  or  the  community  has  made  it 
really  hard  and  for  companies  to  wonder  about  why  even  bother  about  high  maturity  when  low 
maturity  can  deliver  good  repeatable  processes.  Fewer  companies  now  even  want  to  pursue 
ML4/ML5  and  its  drawing  more  criticism  and  negative  attitude  towards  High  Maturity. 

Organizations  tell  us  that  it  seems  that  the  disconnect  between  the  theory  and  practice  is  growing 
larger  and  wider  with  all  these  new  concepts,  PhD  paper  type  approach,  and  theory  out  there, 
which  is  changing  by  the  hour. 

-  Develop  PPMs  according  their  needs 

-  Prepare  internal  people  to  manage  HM  concepts 

Rapid  organizational  change  has  limited  the  knowledge  and  use  of  valuable  techniques  and 
models  that  were  successfully  piloted  but  not  deployed  throughout  the  new  organization. 
Knowledge  of  what  is  possible. 
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In  Conclusion:  What  kinds  of  special  difficulties,  if  any,  related  to  measurement  and 
analysis  have  you  experienced  in  appraising  or  coaching  organizations  that  aspire  to  high 
maturity  status?  (Please  describe  fully) 

If  they  are  serious,  none 

If  not  serious,  then  nothing  works  :-( 

Economy  has  shut  down  the  effort  in  one  company  who  was  intending  to  move  from  L2  to  L4 
capability 

Organizations  getting  too  focused  on  statistical  techniques  rather  than  on  their  useful  application! 
Organizations  that  produce  wonderful  control  charts  that  aren't  used  to  manage  -  or  even  to 
understand  -  project  activities. 

As  requested,  this  is  based  on  my  last  HM  appraisal  which  was  a  real  success  story.  I  have  also 
worked  with  several  other  companies  that  have  not  yet  chosen  to  attempt  a  HM  appraisal,  even 
though  they  wanted  to.  These  other  companies  are  concerned  about  what  is  required  to  be  in  place 
and  have  not  yet  been  willing/able  to  apply  sufficient  resources  to  do  this.  Based  on  my 
understanding  of  the  current  ML4/%  requirements  (based  on  oral  exam,  HM  audit,  etc.),  I  am 
not  willing  to  perform  a  HM  appraisal  unless  the  company  is  as  dedicated  as  this  company  is.  I 
try  to  objectively  explain  what  is  required  and  let  the  company  decide  if  they  want  to  proceed  or 
not.  I  am  usually  not  willing  to  work  with  a  company  for  a  HM  appraisal  unless  they  have  a  full¬ 
time  statistician  and  that  person  have  been  to  the  SEI  HM  classes.  I  Don’t  know  any  other 
reasonable  way  for  a  company  to  understand  what  is  required. 

a)  Misunderstanding  of  predictive  modeling  and  how  it  can  be  used  to  predict  interim  results 

b)  Misunderstanding  of  how  to  record  and  apply  metrics  against  (sub)process  variations  and  use 
this  data  to  select  processes  to  deliver  the  desired  project  objectives.  They  had  very  few  process 
variations  tagged  with  baselines  and  models.  Process  tailoring  was  no  better  than  at  ML  3. 

While  there  is  business  value  in  HM,  the  total  cost  of  implementing  it  has  far  exceeded  financial 
benefits  (currently  this  organization  is  only  focused  on  profit  -  and  it  is  far  easier  and  quicker  to 
simply  reduce  the  workforce  indiscriminately  than  conduct  analysis  and  modeling) 

Nothing  as  such. 

We  do  need  more  comprehensive  information  on  how  to  build  a  useful  PPMs.  Currently,  the 
information  is  in  pieces  and  examples  are  incomplete. 

The  belief  that  most  of  the  PPM  work  is  theoretical  (research  based)  than  being  useful  for  day-to- 
day  project  execution. 

-  understanding  the  right  statistical  tools  to  be  used  in  a  specific  circumstance 

-  achieving  stable  processes  in  complex  environments 

-  finding  individuals  with  the  right  statistical  knowledge  (Six  Sigma  education  most  often  not 
sufficient) 

-  understanding  measurement  error  and  its  impact 

-  understanding  that  estimation  models  are  not  CL/ML4  :o) 

It  may  sound  silly,  but  the  greatest  difficulty  that  I  have  personally  experienced  overall  with  many 
organizations  that  I  work  with  is  the  basic  lack  of  understanding  the  concept  of  “focusing  on 
process  performance  at  the  process  level”  vs  the  results  of  executing  a  process  (project  activity). 
Moving  from  rear  view  mirror  project  management  to  managing  a  project  at  the  process  level  has 
been  an  issue  from  day  one  and  continues  to  be  misunderstood  in  our  industry! 

Inertia  from  several  perspectives 

-  the  belief  that  any  model  is  a  process  model 

-  that  any  measure  is  good  as  a  process  measure 

-  that  MA  SG  1  can  be  ignored 

-  that  the  informative  material  can  be  ignored 
and  others 

1 .  Data  granularity  tends  to  be  a  problem 


116  |  CMU/SEI-2010-TR-022 


In  Conclusion:  What  kinds  of  special  difficulties,  if  any,  related  to  measurement  and 
analysis  have  you  experienced  in  appraising  or  coaching  organizations  that  aspire  to  high 
maturity  status?  (Please  describe  fully) 

2.  Data  validity  is  sometimes  a  question  mark 

3.  Data  may  not  be  available  at  the  sub-process  level 

4.  Understanding  of  statistics  is  poor 

5.  Senior  management  is  in  a  hurry  to  achieve  a  rating  and  therefore  disinclined 
to  wait  for  adequate  data 

In  general  management  doesn't  understand  it  and  the  SEI  does  not  explain  it  well. 

1 .  Lack  of  integrated  tools  for  data  collection 

2.  Lack  of  statistical  expertise  to  understand  the  benefits  of  key  statistical  tools  while  constructing 
the  Process  performance  models  and  baselines 

3.  Dissemination  effort  for  high  maturity  practices  knowledge  and  competencies 

In  general,  the  lack  of  mathematical  skills  and  wanting  a  recipe  to  achieve  maturity  level  4  or  5. 

In  software,  particularly  in  North  America,  a  large  number  of  software  specialists  come  from 
Computer  Science  where  there  is  little  focus  on  mathematics  and  physics.  In  other  countries, 
software  specialists  must  be  engineers,  and  this  greatly  facilitates  implementation  of  high 
maturity  practices  as  described  in  the  CMMI. 

Also,  in  software  organizations,  the  concepts  of  industrial  engineering  and  operations  research  are 
not  well  known.  Process  improvement  is  just  a  new  name  for  disciplines  that  have  existed  for 
many  years  in  other  areas. 

In  addition,  software  is  still  largely  considered  an  art  by  many,  and  not  so  much  an  engineering 
discipline.  Those  who  consider  software  to  be  an  art  sometimes  have  a  point,  and  high  maturity 
practices  as  described  in  the  CMMI  do  not  address  their  concerns.  As  usual,  the  truth  probably 
lies  somewhere  in  the  middle.  A  talented  architect  is  liable  to  design  a  beautiful  building  which  is 
impossible  to  build  and  to  maintain.  A  talented  civil  engineer  is  liable  to  build  an  ugly  square  box 
that  will  withstand  an  earthquake  but  in  which  nobody  wants  to  live.  The  same  applies  to 
software. 

Some  of  the  QPM  PA  is  a  bit  cryptic  for  those  who  haven't  worked  with  it  extensively.  The 
engineering  mindset  does  not  like  grey,  or  “it  depends.”  The  application  of  the  informative 
material  makes  this  difficult.  I  have  to  constantly  remind  them  that  the  informative  material  is  to 
exhibit  intent  of  the  model  -  it  isn't  a  checklist. 

Engineering  processes  do  not  easily  lend  themselves  to  traditional  SPC  techniques  in  large  part 
due  to  duration  of  tasks  and  frequency  and  relevance  of  data  available... 

Almost  a  complete  lack  of  understanding  of  what  measurement  is.  This  goes  beyond  lack  of 
understanding  of  high  maturity  concepts  and  extends  into  the  MA  process  area.  The  majority  of 
organizations  I  have  been  asked  to  coach  have  absolutely  no  concept  of  an  operational  definition 
for  a  measure  and  are  unable  to  construct  a  clear,  unambiguous  procedure  for  collecting,  storing 
or  analyzing  data.  In  practice,  I  have  found  that  these  organizations  are  extremely  sloppy  in  their 
data  collection,  rarely  following  their  procedures,  and  consequently  their  data  is  often  invalid. 
There  is  virtually  no  attention  to  the  measurement  system  itself,  further  undermining  the  integrity 
of  the  data  gathered.  Having  worked  now  with  over  1 5  such  organizations,  all  of  which  had 
previously  been  appraised  at  Level  2  or  Level  3, 1  am  becoming  convinced  that  the  biggest  (if  not 
the  most  visible)  problem  we  face  is  not  in  the  high  maturity  domain,  but  actually  with  the 
fundamental  concepts  of  measurement.  It  should  be  noted  that  the  15  organizations  cited  above 
represent  appraisals  done  by  12  different  lead  appraisers,  so  this  is  not  the  result  of  the  work  of 
one  or  two  bad  apples.  In  nine  of  the  1 5  cases  we  had  to  completely  rework  their  measurement 
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In  Conclusion:  What  kinds  of  special  difficulties,  if  any,  related  to  measurement  and 
analysis  have  you  experienced  in  appraising  or  coaching  organizations  that  aspire  to  high 
maturity  status?  (Please  describe  fully) 

program  —  in  the  other  six,  significant  changes  were  required. 

I  understand  why  the  SEI  audit  program  is  focused  exclusively  on  high  maturity  appraisals,  and  I 
do  not  dispute  the  reasoning.  But  my  experience  indicates  that  the  problem  is  much  bigger  than 
this  and  that  many  people  with  SEI  credentials  who  are  representing  themselves  as  experts  in  the 
model  and  in  measurement  in  particular  are  fundamentally  incompetent  in  these  domains.  This  is 
a  very  troubling  situation,  and  one  that  I  don't  see  being  effectively  addressed  through 
certification  exams. 

Skepticism  of  the  value  of  ML4/5  and  constant  churn  (perceived)  reinterpretation  by  the  SEI  in 
terms  of  what's  required  for  ML4/5. 

The  need  to  collect  contextual  information  and  the  need  to  ensure  that  the  data  collected  is 
accurate. 

Understanding  the  value  of  PPMs  and  PPBs 

Trying  to  quantitatively  manage  percentages  instead  of  actual  variations. 

Helping  them  understand  what  a  sub  process  is  all  about  and  how  to  think  in  tenns  of  controllable 
parameters 

-Too  much  statistical  thinking  not  knowing  the  common-sense  interpretation  of  what  it  means! 

-  Too  much  confusion  due  to  different  interpretations  of  what  high  maturity  means  especially 
after  the  SEI  intervention.  Example:  everyone  is  busy  developing  R2  adjusted  values  and 
ANOVA 

-  not  knowing  that  these  don’t  apply  to  what  they  are  doing!  A  simple  examination  of  a  scatter 
plot  would  reveal  that  there  is  no  relationship  between  two  variables  -  why  would  one  need  a 
correlation  analysis?  Have  you  ever  seen  ANOVA  being  used  to  understand  how  a  set  of  12 
heterogeneous  projects  differ  in  their  execution  characteristics! 

-  Lack  of  focus  on  doing  good  design,  acquiring  good  domain  competencies  and  use  and 
management  of  “tools  and  technology”  to  drive  software  solutions.  Instead,  it  is  most  unfortunate 
that  with  the  current  SEI  emphasis  on  statistical  thinking,  everyone  is  busy  trying  to  “fix” 
software  problems  with  statistics! 

Manage  subprocesses  under  iteration  model 

The  most  common  inhibitors  are  a  lack  of  buy-in  amongst  project  engineers  and  a  sense,  by  this 
same  group,  that  the  nature  of  development  activities  are  too  unique  and  dynamic  to  be  modeled 
and  managed  effectively  using  statistical  techniques  or  modeling  and  simulation.  Additionally,  it 
is  a  great  challenge  to  obtain  finance  (labor  cost)  data  at  the  level  of  granularity  necessary  to 
determine  the  actual  cost  involved  in  performing  various  discrete  engineering  activities. 

It  is  becoming  more  difficult  to  align  the  goals  of  l)achieving  the  high  maturity  rating,  given  the 
evolving  high  maturity  expectations  and  2)  clearly  demonstrating  the  value  to  the  enterprise  of  the 
high  maturity  efforts 

Lack  of  alignment  between  business  objectives  and  project/process  measures. 

Lack  of  contextual  knowledge  when  analyzing  data  (I  found  a  company  showing  a  control  chart 
of  defect  removal  efficiency  in  peer  reviews  with  a  LCL  negative,  what's  the  meaning  of  that?  do 
they  include  new  errors  into  the  reviewed  work  product  when  reviewing?) 

Lack  of  statistical  training,  till  trying  to  represent  a  distribution  by  a  central  tendency  indicator 
lack  of  granularity  in  the  data  for  stratifying  process  and  sub  process  behavior. 

Climbing  the  learning  curve  of  the  metrics/definitions  coupled  with  project/org  goals  and 
objectives  and  their  application  implementation  across  various  lifecycles. 
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In  Conclusion:  What  kinds  of  special  difficulties,  if  any,  related  to  measurement  and 
analysis  have  you  experienced  in  appraising  or  coaching  organizations  that  aspire  to  high 
maturity  status?  (Please  describe  fully ) 

Many  organizations  want  to  become  high  maturity  before  they  have  consistently  obtained 
medium  maturity. 

Not  thinking  ahead  about  what  they  will  need  with  respect  to  measurement  to  determine  if  they 
are  meeting  the  organization's  business  goals  and  objectives.  In  too  many  organizations,  there  is  a 
great  reliance  on  the  examples  in  the  CMMI  book  for  GP  2.8  to  establish  their  measurement 
needs.  There  is  insufficient  attention  paid  when  an  organization  is  at  lower  levels  of  maturity  to 
drilling  down  from  business  goals  and  objectives  to  the  various  process  areas  to  determine  to 
what  extent  these  process  areas  support  the  achievement  of  these  goals  and  objectives. 

Training  or  education  for  SEPG  members  on  the  high  maturity  practices  should  be  provided  with 
examples  of  PP,  PMC,  IPM,  RSKM,  and  other  related  PAs  such  as  the  engineering  PAs.  However 
this  has  not  been  easy  because  1)  SEPG  staffs  do  not  have  enough  time  to  take  the 
training/education,  2)  their  understanding  of  M&A  and  SPC  is  incomplete,  and  3)  they  need  to 
learn  how  to  identify  critical  or  highest  priority  weakness  and  a  possible  improvement  approach. 

In  the  beginning  our  organization  believed  that  we  were  living  both  the  letter  and  spirit  of  the 
CMMI  model.  Now  that  the  SEI  HM  training  is  not  aligned  with  the  goals  and  practices  of  the 
high  maturity  model  the  organization  appears  stalled.  As  an  appraiser  and  coach,  this  is  very 
frustrating.  I  take  great  pains  to  be  practical  and  develop  solutions  that  work  for  the  project  and 
ensure  that  we  are  compliant  with  the  model.  Without  good  examples  from  the  real  world,  it  is 
very  difficult  to  impose  process  steps  and  constraints  that  do  not  benefit  the  organization  just  to 
be  model  compliant.  I  have  heard  from  those  within  SEI  that  they  have  seen  development  projects 
that  successfully  employ  all  the  material  in  their  HM  training  with  great  benefit  to  the 
organization.  However,  when  I  ask  for  examples,  I  am  told  that  they  cannot  be  provided  because 
of  non  disclosure  issues. 

1.  Not  defining  standard  measures  for  the  sub-processes  as  part  of  ML3  journey  and  lack  of 
context  information  on  engineering  process  tailoring 

2.  Availability  of  sufficient  granular  data  and  lack  of  mechanisms  to  collect  such  granular  data 
and  context 

3.  Coming  up  with  prediction  models  that  are  accurate 

Lack  of  statistical  Tools 

Inadequate  statistical  knowledge 

Inadequate  involvement  from  senior  management 

1 .  Controllable  factors  identification,  what  are  good  factors,  and  with  good  quality  historical  data 

2.  No  detailed  process  description  or  measurement  definition,  so  when  they  go  to  statistical 
analysis,  the  variation  is  wide 

3.  General  analysis  method  definition,  doesn't  provide  detail  on  explaining  such  as  the 
relationship  among  different  measures 

For  other . companies  level  3  activities  are  not  enough  institutionalized. 

Variance  of  data  perhaps  due  to  human  factors  and  project  types  make  the  accumulation  of 
enough  historical  data  difficult. 

Personnel  not  believing  that  the  model  “really  wants  the  org  to  control  sub  processes”... 

The  vision  of  what  is  required  for  high  maturity  is  fuzzy  with  v  1.2.  This  is  especially  apparent 
with  appraisal  team  members,  not  all  of  which,  that  have  appropriate  training  in  HM.  So,  it 
becomes  difficult  to  find  consensus  based  on  what  is  required/expected  in  the  model.  And  when 
this  is  fuzzy,  I  think  appraisal  teams  settle  for  somewhat  less  than  what  is  really  expected  for  HM. 
When  I  participated  in  several  HM  appraisals,  it  was  before  I  took  the  high  maturity  training.  I 
had  studied  the  model  and  attended  SEPG/CMMI  Workshop  presentations,  but  had  not  taken  the 
HM  training.  The  HM  training  really  helped  to  understand  what  the  true  vision  of  HM  is.  This 
needs  to  be  translated  into  required  and  expected  components  of  the  CMMI.  There  needs  to  be 
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In  Conclusion:  What  kinds  of  special  difficulties,  if  any,  related  to  measurement  and 
analysis  have  you  experienced  in  appraising  or  coaching  organizations  that  aspire  to  high 
maturity  status?  (Please  describe  fully ) 

some  appropriate  examples  of  “how  much  is  enough”  provided  to  lead  appraisers  to  help  them 
with  this  issue.  Also,  I  think  that  the  concept  of  process  performance  models  is  not  well 
understood.  I  don't  think  the  organizations  I  appraised  (prior  to  HM  training)  took  full  advantage 
of  them. 

Underestimating  time  and  effort  needed  for  effective  modeling  and  its  application  across  projects 

1 .  Getting  the  ATMs  to  understand  the  level  of  statistical  management  that  should  be  done  while 
doing  QPM  and  OPP 

2.  Making  sense  of  the  process  performance  models;  I  have  often  seen  that  some  organizations 
come  up  with  fancy,  statistically  valid  models  but  they  do  not  think  of  the  “story  that  the  model 
tells  us.”  They  come  up  with  factors  which  are  not  really  relevant  at  all  for  a  Project  Manager. 
Hence  the  PM  then  is  at  a  loss  to  understand  what  s/he  should  do  with  the  models. 

3.  Identifying  the  critical  sub  process  based  on  the  PPMs  and  identifying  sub  processes  which 
would  give  them  enough  data  points  to  enable  them  to  do  effective  statistical  management 

ATM  lack  high  maturity  knowledge 

Ensuring  that  the  measurement  definitions  used  in  the  organization  will  produce  the  desired 
results  over  time. 

Recurring  issues  have  included: 

-  judging  the  quality  of  the  data 

-  organizational  cultures  that  keep  measurements  and  measurement  analysis  skills  private  to 
management 

Difficulty  in  understanding  statistical  concepts  and  SPC 

I  am  currently  just  coaching  a  few  organizations  on  HM  practices. 

In  my  impression,  connecting  the  organization's  HM  activities  substantially  to  their  business 
objectives  is  not  always  easy. 

Getting  the  proper  balance  between  dis-aggregating  the  data  and  having  sufficient  data,  and 
defining  the  qualitative  distinctions  needed  to  better  identify  special  causes  of  variation. 

Upper  stream  phases  of  the  project  are  rather  difficult  to  measure  and  analyze  in  a  real  time 
manner. 

Anyway  the  target  organization's  process  improvement  for  PPB  and  PPM  based  quantitative  and 
statistical  management  is  still  under  way,  and  another  appraisal,  SCAMPI  A  or  mini-appraisal,  is 
planned  a  year  or  more  ahead. 

Therefore  most  of  the  answers  above  on  PPM's  and  PPB's  are  based  on  the  as-is  status  of  them, 
that  have  not  always  fully  implemented  the  CMMI  HM  practices. 

And  unfortunately  I  had  little  time  for  additional  document  reviews  or  interviews  for  this 
feedback.  Therefore  these  answers  are  based  on  my  own  understandings  as  they  are,  that  could  be 
updated  anew  if  I  could  have  additional  objective  evidences. 

-  make  people  understand  what  is  expected  by  an  analysis 

-  make  people  understand  the  concept  of  PPM  and  PPB 

-  help  companies  identify  which  PPMs  are  useful  to  their  business  and  find  the  relationship 
between  the  attributes 

Some  organization  and  appraisal  team  members,  especially  those  with  extensive  Six  Sigma 
backgrounds,  lack  understanding  of  or  appreciation  for  process  performance  models.  They 
engage  in  arguments  that  defer  meaningful  discussion  and  learning. 

Most  businesses  I've  worked  with  share  similar  thoughts  and  experiences.  The  SEI  is  making 
assumptions  &  interpretations  regarding  Quantitative  Management,  SPC  etc...  which  are 
traditionally  based  on  manufacturing  standards  and  are  not  allowing  organizations  to  perform 
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maturity  status?  (Please  describe  fully) 

their  own  interpretations  to  best  utilize  QM,  SPC  etc...  methodologies  in  a  Systems/Software 
enviromnent  that  benefit  the  organization. 

Lack  of  integration  of  data  collection,  analysis,  modeling,  and  reporting  tools. 

Failure  to  use  process  performance  models  and  baselines  in  the  initial  contract  bid  process. 

I  have  found  many  organizations  that  have  previously  been  appraised  at  ML4  or  ML5  that  should 
not  have  been.  Explaining  this  level  of  inconsistency  is  an  on-going  challenge. 

Every  project  thinks  it  is  so  different  that  it  can't  leam  from  other  projects'  data. 

They  don't  usually  understand  the  details  of  setting  up  PPO,  PPB,  PPM,  etc. 

People  often  believe  they  understand  high  maturity  concepts  and  the  intent  of  CMMI  practices. 
They  later  find  that  they  are  missing  key  points  and  have  to  adjust  their  thinking  and  their  efforts. 
This  is  frustrating  to  them  because  it  generally  results  in  everything  taking  longer  than  originally 
planned. 

same  as  above 

Not  enough  coaching  and  mentoring,  from  determining  critical  business  objective,  to  defining  and 
using  value  added  PPB  and  PPM. 

Same  as  above 

Knowledge  of  what  is  possible. 

The  org  used  in  this  survey  will  go  for  ML  4  appraisal  in  June  2009.  Two  kinds  of  difficulties  are 
encountered: 

1 .  Initially,  to  properly  set  and  link  the  business  goals,  QPPOs,  baselines,  models,  key  process 
elements,  indicators,  and  needed  measures.  It  is  hard  to  go  back  if  they  are  not  thought  out  in  the 
beginning. 

2.  The  needed  infrastructure  supports. 
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Appendix  C  Analysis  Methods  Used  in  this  Report 


Summarizing  the  Results 

Most  of  the  results  described  in  this  report  summarize  relationships  between  two  variables.  As 
seen  beginning  with  Figure  6-1  on  page  47,  many  of  them  are  described  using  a  graphical  mosaic 
representation  that  shows  in  an  intuitive  visual  manner  the  extent  to  which  the  survey 
respondents’  answers  vary  in  a  consistent  manner.  Figure  6-1  is  repeated  here  for  purpose  of 


Figure  6-1  repeated  for  example:  Relationship  between  emphasis  on  healthy  process  performance 
model  ingredients  and  overall  value  attributed  to  process  performance  models 

The  values  for  each  x-variable  are  displayed  along  the  horizontal  axis  of  the  mosaic,  and  the 
variable  name  is  displayed  below  it.  Labels  for  the  respondents’  answers  to  the  y-factor  are 
displayed  to  the  right  of  the  mosaic  on  the  vertical  axis;  the  name  of  the  y-factor  used  in  the 
statistical  software  is  shown  on  the  left  side  of  the  mosaic.  The  proportions  of  responses  for  each 
category  of  the  x-variable  are  shown  in  separate  columns  of  the  mosaic,  where  each  value  of  the 
y-variable  is  represented  in  a  separate  mosaic  tile.  The  percentages  represented  by  0  to  1  also 
appear  to  the  left  of  the  full  mosaic  and  correspond  to  the  heights  of  the  tiles.  A  legend  to  the  right 
shows  the  possible  values  of  the  y-variable. 

Notice  also  that  the  width  of  each  column  varies  in  proportion  to  the  number  of  responses  for  each 
category  of  the  x-variable.  This  can  provide  a  quick  sense  of  how  evenly  or  unevenly  the  survey 
answers  are  distributed. 
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The  overall  strength  of  the  relationship  between  the  two  variables  can  be  described  by  the  value  of 
the  gamma  statistic.  Goodman  and  Kruskal's  gamma  (y)  is  an  ordinal  measure  of  association  that 
is  appropriate  for  ordered  categorical  measures  such  as  those  that  are  used  in  this  report.29  It  is 
symmetric,  which  means  that  its  value  will  be  the  same  regardless  of  which  variable  is  considered 
to  be  an  x-variable  or  a  y-variable.  The  value  of  gamma  represents  the  difference  between 
concordant  and  discordant  pairs  of  values  on  two  variables.  It  is  computed  as  the  excess  of 
concordant  pairs  as  a  percentage  of  all  pairs,  ignoring  ties.  The  notion  of  concordance  for  any  pair 
of  values  means  that  as  the  x  value  increases  its  corresponding  y  value  also  must  increase  (or 
decrease  for  negative  relationships).  Gamma  is  based  on  weak  monotonicity  (i.e.,  ignoring  ties 
means  that  the  y- value  can  remain  the  same  rather  than  increase). 

Similar  to  many  other  correlation  coefficients  and  measures  of  association,  gamma  varies  from  -1 
to  1.  Values  of  0  usually  indicate  statistical  independence  (no  relationship)  and  values  of  1 
indicate  almost  perfect  relationships  (-1  is  an  almost  perfect  negative  relationship,  where  values 
for  one  variable  decrease  while  the  other  increases).  Gamma  is  a  proportional  reduction  in  error 
(PRE)  statistic  with  an  intuitive  interpretation.  Conceptually  analogous  to  Pearson’s  r2  for  interval 
or  ratio  data,  the  value  of  gamma  is  the  proportion  of  paired  comparisons  where  knowing  the  rank 
order  of  one  variable  reduces  the  proportionate  error  in  predicting  the  rank  order  of  the  other 
variable.  So,  for  example,  if  gamma  is  .75  then  knowing  the  independent  variable  reduces  our 
error  in  predicting  the  rank  of  the  dependent  variable  by  75  percent.  In  this  sense,  gamma  is  a 
measure  of  relative  accuracy. 

Composite  Measures 

Many  of  the  relationships  described  in  this  report  use  composite  measures  that  are  based  on 
combinations  of  several  related  component  questions.  The  measure  of  emphasis  on  “healthy 
process  performance  model  ingredients”  shown  in  Figure  6- Ion  page  47  and  repeated  on  the 
previous  page  is  one  such  composite.  As  noted  there,  it  is  based  on  the  appraisers’  answers  to  a 
group  of  related  questions  about  the  emphasis  that  the  organizations  put  on  the  healthy  ingredients 
of  process  performance  models  The  possible  answers  to  those  questions  included  “extensive,” 
“substantial,”  “moderate,”  “limited”  and  “little  if  any.”30 

Like  most  of  the  other  composite  measures  used  in  this  report,  the  one  in  Figure  6-1  is  a  weighted, 
summed  index  of  the  respondents’  answers  to  each  of  the  questions.31  Much  like  a  grade-point 
average,  the  answers  are  assigned  ordered  numeric  values  that  are  added  and  then  divided  by  the 
number  of  valid  answers  to  the  series  of  questions  for  each  respondent.32  For  example  in  Figure 
6-1,  “extensive”  answers  are  scored  as  the  value  5,  “substantial”  as  4,  down  to  “little  if  any”  as  1. 


29  A  clear  description  can  be  found  in  Linton  Freeman’s  now  classic  text  [33], 

30  The  very  few  answers  of  “Don't  know”  and  “does  not  apply”  were  excluded  from  the  calculations.  Answers  to  the 
“other"  categories  that  sometimes  exist  in  related  question  sets  also  were  included  in  the  composite  indices. 

31  In  other  instances,  the  composite  variables  were  simply  counts  of  the  numbers  of  check  boxes  the  respondents 
selected. 

32  The  weighting  and  summing  are  mathematically  equivalent  to  an  arithmetic  mean;  however,  also  much  like  a 
grade  point  average,  the  results  are  rank  orders.  Such  indices  are  not  interval-  or  ratio-level  measures  that  can 
be  added  or  multiplied  meaningfully. 
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Hence  the  values  on  the  extremes  of  the  weighted  sum  measures  require  consistency  of  replies 
across  all  of  their  respective  component  questions. 

The  index  scores  are  separated  into  the  categories  shown  on  the  figures’  x  or  y  axes  based  on  the 
distribution  of  the  response  values.  The  category  cutting  points  are  set  based  on  their  closeness  to 
the  component  questions’  response  category  values  and  ensuring  that  there  are  enough  cases  in 
each  category  for  ease  of  interpretation.  In  Figure  6-1,  the  lowest  category  (“<  Moderate”) 
includes  the  composite  scores  with  values  less  than  3.  The  second  category  (“Moderate  to  < 
midway  toward  substantial”)  includes  values  that  range  from  3  to  less  than  3.5.  The  third  category 
(“Toward  but  <  substantial”)  ranges  from  3.5  to  less  than  4.  The  highest  category  (“Substantial  or 
better”)  includes  composite  scores  that  are  equal  to  or  greater  than  4. 

There  are  several  reasons  to  combine  the  component  questions  into  single  composite  indices.  Of 
course,  reducing  the  number  simplifies  visual  interpretation.  While  it  may  seem  counterintuitive, 
combining  the  components  also  follows  a  basic  reliability  principle.  There  always  is  noise  in 
survey  data  (actually  in  measured  data  of  any  kind).  Respondents  can  be  uncertain  about  their 
answers  concerning  the  details  of  a  specific  question,  or  the  lack  of  clarity  in  the  wording  of  a 
specific  question  may  cause  different  respondents  to  attribute  different  meanings  to  the  same 
question.  Other  things  being  equal,  the  unreliability  can  be  averaged  such  that  the  composite 
index  is  more  reliable  than  many  or  all  of  its  individual  components  [22]  [23]  [24], 

Interpretation 

Survey  data  such  as  these  do  not  speak  for  themselves.  Interpretation  is  necessary  for  all  statistical 
analyses,  including  those  based  on  controlled  experiments.  Perceptions  and  expectations  often 
differ  among  survey  respondents  and  maturity  levels.  Moreover,  survey  data  such  as  these  often 
are  collected  at  a  single  point  in  time.  It  is  difficult  to  separate  cause  and  effect,  which  often  are 
reciprocal  over  time.  Proportions  and  strength  of  association  sometimes  vary  in  subtle  ways.  Still, 
the  differences  described  in  this  report  are  consistent  with  what  we  think  we  know  about 
capability  maturity  as  well  as  the  benefits  that  are  possible  through  the  suitable  use  of 
measurement  and  analysis. 


125  |  CMU/SEI-2010-TR-022 


126  |  CMU/SEI-2010-TR-022 


Bibliography 


[1]  D.  Goldenson,  J.  McCurley,  and  R.  W.  Stoddard,  "Use  and  Organizational  Effects  of 
Measurement  and  Analysis  in  High  Maturity  Organizations:  Results  from  the  2008  SEI  State 
of  the  Measurement  and  Analysis  Practice  Surveys,"  Carnegie  Mellon  Software  Engineering 
Institute  (CMU/SEI-2008-TR-024),  2009.  [Online]. 
http://www.sei.cmu.edu/library/abstracts/reports/08tr024.cfm 

[2]  D.  Goldenson.  (2007,  Nov.)  Understanding  CMMI  Measurement  Capabilities  &  Impact  on 
Perfonnance:  Results  from  the  2007  SEI  State  of  the  Measurement  Practice  Survey. 
Presentation.  [Online]. 

http://www.sei.cmu.edu/library/abstracts/presentations/measurementsurvey.cfm 

[3]  R.  W.  Stoddard  and  D.  Goldenson,  "Approaches  to  Process  Performance  Modeling:  A 
Summary  from  the  SEI  Series  of  Workshops  on  CMMI  High  Maturity  Measurement  and 
Analysis,"  Carnegie  Mellon  Software  Engineering  Institute  CMU/SEI-2009-TR-021,  2010. 
[Online] .  http://www.sei.mcu.edu/library/abstracts/reports/09tr02 1  .cfm 

[4]  M.  Brown  and  D.  Goldenson,  "Measurement  and  Analysis:  What  Can  and  Does  Go 
Wrong?,"  in  Proceedings  of  the  10th  IEEE  International  Symposium  on  Software  Metrics 
(METRICS'04),  Chicago,  2004. 

[5]  A.  Gopal,  M.  Krishnan,  T.  Mukhopadhyay,  and  D.  Goldenson,  "Measurement  Programs  in 
Software  Development:  Determinants  of  Success,"  IEEE  Transactions  on  Software 
Engineering,  vol.  28,  no.  9,  Sep.  2002. 

[6]  D.  Goldenson,  J.  Jarzombek,  and  T.  Rout,  "Measurement  and  Analysis  in  CMMI  Models  and 
Software  Process  Improvement,"  Crosstalk,  vol.  17,  no.  7,  Jul.  2004. 

[7]  K.  El  Emam,  D.  Goldenson,  J.  McCurley,  and  J.  Herbsleb,  "Success  or  Failure?  Modeling  the 
Likelihood  of  Software  Process  Improvement,"  Empirical  Software  Engineering,  vol.  6,  no. 

3,  Sep.  2001. 

[8]  D.  Goldenson,  A.  Gopal,  D.  White,  and  T.  Mukhopadhyay,  "Determininants  of  Success  in 
Software  Measurement  Programs,"  in  Proceedings  of  the  Sixth  International  Software 
Metrics  Symposium  (METRICS'99),  Boca  Raton,  1999. 

[9]  C.  Deephouse,  T.  Mukhopadhyay,  D.  Goldenson,  and  M.  Kellner,  "Software  Processes  and 
Project  Performance,"  Journal  of  Management  Information  Systems,  vol.  12,  no.  3,  pp.  187- 
205,  Dec.  1995. 

[10]  G.  W.  Corder  and  D.  I.  Foreman,  Nonparametric  Statistics  for  Non-Statisticians:  A  Step-by- 
Step  Approach.  Hoboken,  NJ:  Wiley,  2009. 


127  |  CMU/SEI-2010-TR-022 


[11]  H.  B.  Mann  and  D.  R.  Whitney,  "On  a  Test  of  Whether  One  of  Two  Random  Variables  is 
Stochastically  Larger  than  the  Other  Two,"  Annals  of  Mathematical  Statistics,  vol.  18,  no.  1, 
pp.  50-60,  1947. 

[12]  J.  Payne,  "Fishing  Expedition  Probability:  The  Statistics  of  Post  Hoc  Hypothesizing,"  Polity, 
vol.  7,  no.  1,  pp.  130-138,  1974. 

[13]  R.  Young  and  R.  Stoddard.  (2008,  Mar.)  A  Practitioner  View  of  CMMI  Process  Performance 
Models,  SEPGNA.  [Online],  http://www.sei.cmu.edu/sema/presentations/practitioner- 
view.pdf 

[14]  R.  E.  Park,  W.  B.  Goethert,  and  W.  A.  Florae,  "Goal -Driven  Software  Measurement — A 
Guidebook,"  Carnegie  Mellon  Software  Engineering  Institute  CMU/SEI-96-HB-002,  1996. 
[Online],  http://www.sei.cmu.edu/library/abstracts/reports/96b002.cfm 

[15]  R.  van  Solingen  and  E.  Berghout,  The  Goal/Question/Metric  Method:  A  Practical  Guide  for 
Quality  Improvement  of  Software  Development.  Chicago:  McGraw-Hill  Publishers,  1999. 

[16]  D.  Goldenson,  "What  Does  it  Take  to  Succeed  in  (the  Software  Measurement)  Business?," 
Software  Quality,  ASQ  Software  Quality  Newsletter,  Dec.  2000. 

[17]  J.  P.  Elm,  et  al.,  "A  Survey  of  Systems  Engineering  Effectiveness  -  Initial  Results  (with 
detailed  survey  response  data),"  Carnegie  Mellon  Software  Engineering  Institute  CMU/SEI- 
2008-SR-034,  2008.  [Online],  http://www.sei.cmu.edu/library/abstracts/reports/08sr034.cfm 

[18]  R.  W.  Ferguson,  "Systems  Engineering  Complexity  &  Project  Management,"  in  CMMI 
Technology  Conference  &  User  Group,  Denver,  2007.  [Online]. 
http://www.sei.cmu.edu/library/abstracts/presentations/complexity.cfm 

[19]  R.  Young,  R.  Stoddard,  and  M.  Konrad,  "If  You’re  Living  the  ‘High  Life,’  You’re  Living  the 
Informative  Material,"  in  SEPG  NA,  Tampa,  2008.  [Online]. 
http://www.sei.cmu.edu/sema/presentations/highlife.pdf 

[20]  E.  Tufte,  Visual  and  Statistical  Thinking:  Displays  of  Evidence  for  Making  Decisions. 
Graphics  Press,  1997. 

[21]  J.  Tukey ,  Exploratory  Data  Analysis.  Boston:  Addison  Wesley,  1977. 

[22]  T.  Hill  and  P.  Lewicki,  Statistics:  Methods  and  Applications:  a  Comprehensive  Reference. 
StatSoft,  Inc.,  2006. 

[23]  J.  S.  Coleman,  Models  of  Change  and  Response  Uncertainty.  Englewood  Cliffs:  Prentice- 
Hall,  1964. 

[24]  J.  P.  Guilford,  Psychometric  Methods,  2nd  ed.  London:  McGraw-Hill  Inc.,  1954. 

[25]  M.  Agrawal  and  K.  Chari,  "A  Study  of  CMM  Level  5  Projects,"  IEEE  Transactions  of 
Software  Engineering,  vol.  33,  no.  3,  pp.  145-156,  Mar.  2007. 


128  |  CMU/SEI-2010-TR-022 


[26]  D.  J.  Reifer,  "Profiles  of  Level  5  CMMI  Organizations,"  Crosstalk,  vol.  20,  no.  1,  pp.  24-28, 
Jan.  2007. 

[27]  J.  Sutherland,  "Scrum  and  CMMI  Level  5:  The  Magic  Potion  for  Code  Warriors,"  in 
Proceedings  of  the  AGILE  2007  Conference,  Washington,  DC,  2007,  pp.  272-278. 

[28]  P.  J.  McLoone  and  S.  L.  Rohde,  "Performance  Outcomes  of  CMMI-Based  Process 
Improvements,"  Software  Tech  News,  vol.  10,  no.  1,  pp.  5-9,  Mar.  2007. 

[29]  M.  Sapp,  R.  Stoddard,  and  T.  Christian,  "Cost,  Schedule,  and  Quality  Improvements  at 
Warner  Robins  Air  Logistics  Center,"  Software  Tech  News,  vol.  10,  no.  1,  pp.  10-13,  Mar. 
2007. 

[30]  D.  Garmus  and  S.  Iwanicki,  "Improved  Performance  Should  Be  Expected  from  Process 
Improvement,"  Software  Tech  News,  vol.  10,  no.  1,  pp.  14-17,  Mar.  2007. 

[31]  A.  Q.  Liu,  "Motorola  Software  Group's  China  Center:  Value  Added  by  CMMI,"  Software 
Tech  News,  vol.  10,  no.  1,  pp.  18-23,  Mar.  2007. 

[32]  K.  El  Emam,  "TrialStat  Corporation:  On  Schedule  with  High  Quality  and  Cost  Savings  for 
the  Customer,"  Software  Tech  News,  vol.  10,  no.  1,  pp.  24-29,  Mar.  2007. 

[33]  L.  C.  Freeman,  Elementary’  Applied  Statistics  for  Students  in  Behavioral  Science.  New  York: 
John  Wiley  &  Sons,  1965. 

[34]  M.  Kasunic,  "The  State  of  Software  Measurement  Practice:  Results  of  2006  Survey," 
Carnegie  Mellon  Software  Engineering  Institute  CMU/SEI-2006-TR-009,  2006.  [Online]. 
http://www.sei.cmu.edu/library/abstracts/reports/06tr009.cfm 


129  |  CMU/SEI-2010-TR-022 


130  |  CMU/SEI-2010-TR-022 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions, 
searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments 
regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington 
Headquarters  Services,  Directorate  for  information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to 
the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 

1.  AGENCY  USE  ONLY 

(Leave  Blank) 

2.  REPORT  DATE 

June  2010 

3.  REPORT  TYPE  AND  DATES 

COVERED 

Final 

4.  TITLE  AND  SUBTITLE 

Performance  Effects  of  Measurement  and  Analysis:  Perspectives  from  CMMI  High  Maturity 
Organizations  and  Appraisers 

5.  FUNDING  NUMBERS 

FA8721-05-C-0003 

6.  author(s) 

James  McCurley  &  Dennis  R.  Goldenson 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Software  Engineering  Institute 

Carnegie  Mellon  University 

Pittsburgh,  PA  15213 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

CMU/SEI-201 0-TR-022 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

HQ  ESC/XPK 

5  Eglin  Street 

Hanscom  AFB,  MA  01731-21 16 

10.  SPONSORING/MONITORING 

AGENCY  REPORT  NUMBER 

ESC-TR-201 0-022 

11.  SUPPLEMENTARY  NOTES 

12a  distribution/availability  statement 

Unclassified/Unlimited,  DTIC,  NTIS 

12B  DISTRIBUTION  CODE 

1 3.  ABSTRACT  (MAXIMUM  200  WORDS) 

This  report  describes  results  from  two  recent  surveys  conducted  by  the  Software  Engineering  Institute  (SEI)  to  collect  information  about 
the  measurement  and  analysis  activities  of  software  systems  development  organizations.  Representatives  of  organizations  appraised  at 
maturity  levels  4  and  5  completed  the  survey  in  2008.  Using  a  variant  of  the  same  questionnaire  in  2009,  certified  high  maturity  lead 
appraisers  described  the  organizations  that  they  had  most  recently  coached  or  appraised  for  the  achievement  of  similar  high  maturity 
levels.  The  replies  to  both  surveys  were  generally  consistent  even  though  the  two  groups  are  often  thought  to  be  quite  different.  The 
results  of  the  surveys  suggest  that  the  organizations  understood  and  used  CMMI-based  process  performance  modeling  and  related 
aspects  of  measurement  and  analysis  a  great  deal.  Both  the  organizational  respondents  in  2008  and  the  appraisers  in  2009  reported 
that  process  performance  models  were  useful  for  the  organizations. 

The  respondents  in  both  surveys  also  judged  process  performance  modeling  to  be  more  valuable  in  organizations  that  understood  and 
used  measurement  and  analysis  activities  more  frequently  and  provided  organizational  resources  and  management  support.  In  addition, 
results  from  the  2009  survey  of  lead  appraisers  indicate  that  organizations  that  achieved  their  appraised  high  maturity  level  goals  also 
found  measurement  and  analysis  activities  more  useful  than  those  organizations  that  did  not  achieve  their  targets. 

14.  SUBJECT  TERMS  15.  NUMBER  OF  PAGES 

High  maturity,  state  of  the  measurement  practice,  survey  results,  process  performance  143 

modeling,  modeling 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION  OF  18.  SECURITY  CLASSIFICATION  OF  19.  SECURITY  CLASSIFICATION  20.  LIMITATION  OF 

REPORT  THIS  PAGE  OF  ABSTRACT  ABSTRACT 

Unclassified  Unclassified  Unclassified  UL 

NSN  7540-01-280-5500 


Standard  Form  298  (Rev.  2-89)  Prescribed  by  ANSI  Std.  Z39-18 
298-102 


